WIP: Adding "push lexer" functionality.

[Jerome-PS]

Hi,

This is my very first (beta) proposal for adding "push lexer" functionality to flex.
The rationale is since bison has a push mode, adding this functionality allows the whole lexer/parser to yield instead of doing a blocking wait for more input data. This is useful for example in embedded systems, where micro controllers do not have threads, but could benefit for online protocol parsing.

How it works:

some (most) local variables from YY_DECL have been transferred to the yy_guts structure.
YY_DECL can now return YY_STALLED if no data is available to complete the current lexeme.
This happens if yy_fill_buffer is equal to 2 (maybe this should be changed to a #define) and a YY_STALLED (preprocessor) symbol exists.
There is a new yy_append_bytes function that appends bytes to the current input buffer (it can increase its size, but does not replace it like yy_scan_bytes).

Status:

All tests pass. The tests will be adapted so they run also in the push lexer configuration, where that makes sense.

Please give me some feedback on the design decisions I made, and what should be changed (for example usage of a % keyword).

Cheers,
Jérôme.

[westes]

Now that #489 has been merged in, there are file conflicts. Could you sort out how to resolve tyour conflicts and push up those additional changes?

[Jerome-PS]

Head does not seem to compile.
Where can I start from?

[Jerome-PS]

Making all in tests
...
tests/ruleset.am:4: error: bad characters in variable name '*_nr_SOURCES'

[jannick0]

Making all in tests
...
tests/ruleset.am:4: error: bad characters in variable name '*_nr_SOURCES'

The error occurs for VPATH builds only I believe. If you built in the source tree for now, then it should work.

I am currently putting together a list of commits for the building process only. This covers VPATH builds, too. Pushing shortly. Not sure which commit to rebase this on.

[Mightyjo]

ruleset.am looks right in HEAD. It can be regenerated during make and it sounds like that has gone sideways.

It's working for me right now. Can you see where it's being remade in the make output? The * wildcard shouldn't be appearing in _SOURCES variable names so something is going wrong when ruleset.sh runs.

[jannick0]

For a VPATH build ruleset.sh shoud be run in the src test dir, i.e. cd $(srcdir) first (from one of my commits I am working on):

------------------------------ tests/Makefile.am ------------------------------
index 43b0bb8..82b80b0 100755
@@ -413,7 +413,7 @@ RULESETS = \
 	$(srcdir)/yyunput.rules
 
 $(srcdir)/ruleset.am: $(srcdir)/ruleset.sh $(RULESETS)
-	$(SHELL) $(srcdir)/ruleset.sh nr r c99 go > $(srcdir)/ruleset.am
+	$(AM_V_GEN) cd $(srcdir) && $(SHELL) ./ruleset.sh nr r c99 go > ./ruleset.am
 
 include $(srcdir)/ruleset.am

Does this help?

[Mightyjo]

Are you trying to do VPATH builds from a clean git checkout? That probably won't work. The maintainer sources have to be built in the source tree once before all the necessary distro files are available to support VPATH.

You should be able to do VPATH builds from a distro tarball right away.

We build a candidate release tarball in two steps:

1. git clean -xf
2. ./autogen.sh && ./configure && make && make distcheck

The second step produces tarballs ready for distribution or unpacking and VPATH building. If you're in a hurry, just run make once in the source tree and then set up VPATH builds from whatever other directory you like.

The change you proposed above will generally work, I think even during distcheck when srcdir is read-only, because ruleset.am is an EXTRA_DIST file. It'll already be available and up-to-date with its dependencies when you do a VPATH build.