FLEX(1) Cosmopolitan General Commands Manual -*-text-*-
๐๐๐๐
๐ณ๐น๐ฒ๐
, ๐ณ๐น๐ฒ๐
++, ๐น๐ฒ๐
โ fast lexical analyzer generator
๐๐๐๐๐๐๐๐
๐ณ๐น๐ฒ๐
[-๐ณ๐ด๐๐ฏ๐ฑ๐
๐ณ๐ต๐๐ถ๐๐น๐ป๐ฝ๐๐๐๐๐๐+?] [-๐[๐ฎ๐ฒ๐
๐ณ๐บ๐ฟ]] [--๐ต๐ฒ๐น๐ฝ] [--๐๐ฒ๐ฟ๐๐ถ๐ผ๐ป]
[-๐ผoฬฒuฬฒtฬฒpฬฒuฬฒtฬฒ] [-๐pฬฒrฬฒeฬฒfฬฒiฬฒxฬฒ] [-๐sฬฒkฬฒeฬฒlฬฒeฬฒtฬฒoฬฒnฬฒ] [fฬฒiฬฒlฬฒeฬฒ .ฬฒ.ฬฒ.ฬฒ]
๐๐๐๐๐๐๐๐๐๐๐
๐ณ๐น๐ฒ๐
is a tool for generating sฬฒcฬฒaฬฒnฬฒnฬฒeฬฒrฬฒsฬฒ: programs which recognize
lexical patterns in text. ๐ณ๐น๐ฒ๐
reads the given input files, or its
standard input if no file names are given, for a description of a
scanner to generate. The description is in the form of pairs of
regular expressions and C code, called rฬฒuฬฒlฬฒeฬฒsฬฒ. ๐ณ๐น๐ฒ๐
generates as
output a C source file, lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒ, which defines a routine ๐๐๐น๐ฒ๐
().
This file is compiled and linked with the -๐น๐ณ๐น library to produce
an executable. When the executable is run, it analyzes its input
for occurrences of the regular expressions. Whenever it finds one,
it executes the corresponding C code.
๐น๐ฒ๐
is a synonym for ๐ณ๐น๐ฒ๐
. ๐ณ๐น๐ฒ๐
++ is a synonym for ๐ณ๐น๐ฒ๐
-+.
The manual includes both tutorial and reference sections:
๐๐ผ๐บ๐ฒ ๐๐ถ๐บ๐ฝ๐น๐ฒ ๐๐
๐ฎ๐บ๐ฝ๐น๐ฒ๐
๐
๐ผ๐ฟ๐บ๐ฎ๐ ๐ผ๐ณ ๐๐ต๐ฒ ๐๐ป๐ฝ๐๐ ๐
๐ถ๐น๐ฒ
๐๐ฎ๐๐๐ฒ๐ฟ๐ป๐
The extended regular expressions used by ๐ณ๐น๐ฒ๐
.
๐๐ผ๐ ๐๐ต๐ฒ ๐๐ป๐ฝ๐๐ ๐ถ๐ ๐๐ฎ๐๐ฐ๐ต๐ฒ๐ฑ
The rules for determining what has been matched.
๐๐ฐ๐๐ถ๐ผ๐ป๐
How to specify what to do when a pattern is matched.
๐๐ต๐ฒ ๐๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ฒ๐ฑ ๐๐ฐ๐ฎ๐ป๐ป๐ฒ๐ฟ
Details regarding the scanner that ๐ณ๐น๐ฒ๐
produces; how to control
the input source.
๐๐๐ฎ๐ฟ๐ ๐๐ผ๐ป๐ฑ๐ถ๐๐ถ๐ผ๐ป๐
Introducing context into scanners, and managing "mini-scanners".
๐๐๐น๐๐ถ๐ฝ๐น๐ฒ ๐๐ป๐ฝ๐๐ ๐๐๐ณ๐ณ๐ฒ๐ฟ๐
How to manipulate multiple input sources; how to scan from strings
instead of files.
๐๐ป๐ฑ-๐ผ๐ณ-๐
๐ถ๐น๐ฒ ๐๐๐น๐ฒ๐
Special rules for matching the end of the input.
๐๐ถ๐๐ฐ๐ฒ๐น๐น๐ฎ๐ป๐ฒ๐ผ๐๐ ๐๐ฎ๐ฐ๐ฟ๐ผ๐
A summary of macros available to the actions.
๐๐ฎ๐น๐๐ฒ๐ ๐๐๐ฎ๐ถ๐น๐ฎ๐ฏ๐น๐ฒ ๐๐ผ ๐๐ต๐ฒ ๐๐๐ฒ๐ฟ
A summary of values available to the actions.
๐๐ป๐๐ฒ๐ฟ๐ณ๐ฎ๐ฐ๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐๐ฎ๐ฐ๐ฐ
Connecting flex scanners together with yacc(1) parsers.
๐๐ฝ๐๐ถ๐ผ๐ป๐
๐ณ๐น๐ฒ๐
command-line options, and the โ%optionโ directive.
๐๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ ๐๐ผ๐ป๐๐ถ๐ฑ๐ฒ๐ฟ๐ฎ๐๐ถ๐ผ๐ป๐
How to make scanners go as fast as possible.
๐๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ถ๐ป๐ด ๐++ ๐๐ฐ๐ฎ๐ป๐ป๐ฒ๐ฟ๐
The (experimental) facility for generating C++ scanner classes.
๐๐ป๐ฐ๐ผ๐บ๐ฝ๐ฎ๐๐ถ๐ฏ๐ถ๐น๐ถ๐๐ถ๐ฒ๐ ๐๐ถ๐๐ต ๐๐ฒ๐
๐ฎ๐ป๐ฑ ๐๐๐๐๐
How ๐ณ๐น๐ฒ๐
differs from AT&T UNIX ๐น๐ฒ๐
and the POSIX ๐น๐ฒ๐
standard.
๐
๐ถ๐น๐ฒ๐
Files used by ๐ณ๐น๐ฒ๐
.
๐๐ถ๐ฎ๐ด๐ป๐ผ๐๐๐ถ๐ฐ๐
Those error messages produced by ๐ณ๐น๐ฒ๐
(or scanners it generates)
whose meanings might not be apparent.
๐๐ฒ๐ฒ ๐๐น๐๐ผ
Other documentation, related tools.
๐๐๐๐ต๐ผ๐ฟ๐
Includes contact information.
๐๐๐ด๐
Known problems with ๐ณ๐น๐ฒ๐
.
๐๐๐๐ ๐๐๐๐๐๐ ๐๐๐๐๐๐๐๐
First some simple examples to get the flavor of how one uses ๐ณ๐น๐ฒ๐
.
The following ๐ณ๐น๐ฒ๐
input specifies a scanner which whenever it
encounters the string "username" will replace it with the user's
login name:
%%
username printf("%s", getlogin());
By default, any text not matched by a ๐ณ๐น๐ฒ๐
scanner is copied to the
output, so the net effect of this scanner is to copy its input file
to its output with each occurrence of "username" expanded. In this
input, there is just one rule. "username" is the pฬฒaฬฒtฬฒtฬฒeฬฒrฬฒnฬฒ and the
"printf" is the aฬฒcฬฒtฬฒiฬฒoฬฒnฬฒ. The "%%" marks the beginning of the rules.
Here's another simple example:
%{
int num_lines = 0, num_chars = 0;
%}
%%
\n ++num_lines; ++num_chars;
. ++num_chars;
%%
main()
{
yylex();
printf("# of lines = %d, # of chars = %d\n",
num_lines, num_chars);
}
This scanner counts the number of characters and the number of
lines in its input (it produces no output other than the final
report on the counts). The first line declares two globals,
"num_lines" and "num_chars", which are accessible both inside
๐๐๐น๐ฒ๐
() and in the ๐บ๐ฎ๐ถ๐ป() routine declared after the second "%%".
There are two rules, one which matches a newline ("\n") and increโ
ments both the line count and the character count, and one which
matches any character other than a newline (indicated by the "."
regular expression).
A somewhat more complicated example:
/* scanner for a toy Pascal-like language */
%{
/* need this for the call to atof() below */
#include <math.h>
%}
DIGIT [0-9]
ID [a-z][a-z0-9]*
%%
{DIGIT}+ {
printf("An integer: %s (%d)\n", yytext,
atoi(yytext));
}
{DIGIT}+"."{DIGIT}* {
printf("A float: %s (%g)\n", yytext,
atof(yytext));
}
if|then|begin|end|procedure|function {
printf("A keyword: %s\n", yytext);
}
{ID} printf("An identifier: %s\n", yytext);
"+"|"-"|"*"|"/" printf("An operator: %s\n", yytext);
"{"[^}\n]*"}" /* eat up one-line comments */
[ \t\n]+ /* eat up whitespace */
. printf("Unrecognized character: %s\n", yytext);
%%
main(int argc, char *argv[])
{
++argv; --argc; /* skip over program name */
if (argc > 0)
yyin = fopen(argv[0], "r");
else
yyin = stdin;
yylex();
}
This is the beginnings of a simple scanner for a language like Pasโ
cal. It identifies different types of tฬฒoฬฒkฬฒeฬฒnฬฒsฬฒ and reports on what
it has seen.
The details of this example will be explained in the following secโ
tions.
๐
๐๐๐๐๐ ๐๐
๐๐๐ ๐๐๐๐๐ ๐
๐๐๐
The ๐ณ๐น๐ฒ๐
input file consists of three sections, separated by a line
with just "%%" in it:
definitions
%%
rules
%%
user code
The dฬฒeฬฒfฬฒiฬฒnฬฒiฬฒtฬฒiฬฒoฬฒnฬฒsฬฒ section contains declarations of simple nฬฒaฬฒmฬฒeฬฒ defiโ
nitions to simplify the scanner specification, and declarations of
sฬฒtฬฒaฬฒrฬฒtฬฒ cฬฒoฬฒnฬฒdฬฒiฬฒtฬฒiฬฒoฬฒnฬฒsฬฒ, which are explained in a later section.
Name definitions have the form:
name definition
The "name" is a word beginning with a letter or an underscore (โ_โ)
followed by zero or more letters, digits, โ_โ, or โ-โ (dash). The
definition is taken to begin at the first non-whitespace character
following the name and continuing to the end of the line. The defโ
inition can subsequently be referred to using "{name}", which will
expand to "(definition)". For example:
DIGIT [0-9]
ID [a-z][a-z0-9]*
This defines "DIGIT" to be a regular expression which matches a
single digit, and "ID" to be a regular expression which matches a
letter followed by zero-or-more letters-or-digits. A subsequent
reference to
{DIGIT}+"."{DIGIT}*
is identical to
([0-9])+"."([0-9])*
and matches one-or-more digits followed by a โ.โ followed by zero-
or-more digits.
The rฬฒuฬฒlฬฒeฬฒsฬฒ section of the ๐ณ๐น๐ฒ๐
input contains a series of rules of
the form:
pattern action
The pattern must be unindented and the action must begin on the
same line.
See below for a further description of patterns and actions.
Finally, the user code section is simply copied to lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒ verbaโ
tim. It is used for companion routines which call or are called by
the scanner. The presence of this section is optional; if it is
missing, the second "%%" in the input file may be skipped too.
In the definitions and rules sections, any indented text or text
enclosed in โ%{โ and โ%}โ is copied verbatim to the output (with
the %{}'s removed). The %{}'s must appear unindented on lines by
themselves.
In the rules section, any indented or %{} text appearing before the
first rule may be used to declare variables which are local to the
scanning routine and (after the declarations) code which is to be
executed whenever the scanning routine is entered. Other indented
or %{} text in the rule section is still copied to the output, but
its meaning is not well-defined and it may well cause compile-time
errors (this feature is present for POSIX compliance; see below for
other such features).
In the definitions section (but not in the rules section), an uninโ
dented comment (i.e., a line beginning with "/*") is also copied
verbatim to the output up to the next "*/".
๐๐๐๐๐๐๐๐
The patterns in the input are written using an extended set of regโ
ular expressions. These are:
x Match the character โxโ.
. Any character (byte) except newline.
[xyz] A "character class"; in this case, the pattern matches
either an โxโ, a โyโ, or a โzโ.
[abj-oZ] A "character class" with a range in it; matches an โaโ, a
โbโ, any letter from โjโ through โoโ, or a โZโ.
[^A-Z] A "negated character class", i.e., any character but
those in the class. In this case, any character EXCEPT
an uppercase letter.
[^A-Z\n] Any character EXCEPT an uppercase letter or a newline.
r* Zero or more r's, where โrโ is any regular expression.
r+ One or more r's.
r? Zero or one r's (that is, "an optional r").
r{2,5} Anywhere from two to five r's.
r{2,} Two or more r's.
r{4} Exactly 4 r's.
{name} The expansion of the "name" definition (see above).
"[xyz]\"foo"
The literal string: [xyz]"foo.
\X If โXโ is an โaโ, โbโ, โfโ, โnโ, โrโ, โtโ, or โvโ, then
the ANSI-C interpretation of โ\Xโ. Otherwise, a literal
โXโ (used to escape operators such as โ*โ).
\0 A NUL character (ASCII code 0).
\123 The character with octal value 123.
\x2a The character with hexadecimal value 2a.
(r) Match an โrโ; parentheses are used to override precedence
(see below).
rs The regular expression โrโ followed by the regular
expression โsโ; called "concatenation".
r|s Either an โrโ or an โsโ.
r/s An โrโ, but only if it is followed by an โsโ. The text
matched by โsโ is included when determining whether this
rule is the "longest match", but is then returned to the
input before the action is executed. So the action only
sees the text matched by โrโ. This type of pattern is
called "trailing context". (There are some combinations
of r/s that ๐ณ๐น๐ฒ๐
cannot match correctly; see notes in the
BฬฒUฬฒGฬฒSฬฒ section below regarding "dangerous trailing
context".)
^r An โrโ, but only at the beginning of a line (i.e., just
starting to scan, or right after a newline has been
scanned).
r$ An โrโ, but only at the end of a line (i.e., just before
a newline). Equivalent to "r/\n".
Note that ๐ณ๐น๐ฒ๐
's notion of "newline" is exactly whatever
the C compiler used to compile ๐ณ๐น๐ฒ๐
interprets โ\nโ as.
<s>r An โrโ, but only in start condition โsโ (see below for
discussion of start conditions).
<s1,s2,s3>r
The same, but in any of start conditions s1, s2, or s3.
<*>r An โrโ in any start condition, even an exclusive one.
<<EOF>> An end-of-file.
<s1,s2><<EOF>>
An end-of-file when in start condition s1 or s2.
Note that inside of a character class, all regular expression operโ
ators lose their special meaning except escape (โ\โ) and the charโ
acter class operators, โ-โ, โ]โ, and, at the beginning of the
class, โ^โ.
The regular expressions listed above are grouped according to
precedence, from highest precedence at the top to lowest at the
bottom. Those grouped together have equal precedence. For examโ
ple,
foo|bar*
is the same as
(foo)|(ba(r*))
since the โ*โ operator has higher precedence than concatenation,
and concatenation higher than alternation (โ|โ). This pattern
therefore matches eฬฒiฬฒtฬฒhฬฒeฬฒrฬฒ the string "foo" oฬฒrฬฒ the string "ba" folโ
lowed by zero-or-more r's. To match "foo" or zero-or-more "bar"'s,
use:
foo|(bar)*
and to match zero-or-more "foo"'s-or-"bar"'s:
(foo|bar)*
In addition to characters and ranges of characters, character
classes can also contain character class eฬฒxฬฒpฬฒrฬฒeฬฒsฬฒsฬฒiฬฒoฬฒnฬฒsฬฒ. These are
expressions enclosed inside โ[:โ and โ:]โ delimiters (which themโ
selves must appear between the โ[โ and โ]โ of the character class;
other elements may occur inside the character class, too). The
valid expressions are:
[:alnum:] [:alpha:] [:blank:]
[:cntrl:] [:digit:] [:graph:]
[:lower:] [:print:] [:punct:]
[:space:] [:upper:] [:xdigit:]
These expressions all designate a set of characters equivalent to
the corresponding standard C ๐ถ๐๐๐๐() function. For example,
[:alnum:] designates those characters for which isalnum(3) returns
true - i.e., any alphabetic or numeric. Some systems don't provide
isblank(3), so ๐ณ๐น๐ฒ๐
defines [:blank:] as a blank or a tab.
For example, the following character classes are all equivalent:
[[:alnum:]]
[[:alpha:][:digit:]]
[[:alpha:]0-9]
[a-zA-Z0-9]
If the scanner is case-insensitive (the -๐ถ flag), then [:upper:]
and [:lower:] are equivalent to [:alpha:].
Some notes on patterns:
- A negated character class such as the example "[^A-Z]" above
will match a newline unless "\n" (or an equivalent escape
sequence) is one of the characters explicitly present in the
negated character class (e.g., "[^A-Z\n]"). This is unlike how
many other regular expression tools treat negated character
classes, but unfortunately the inconsistency is historically
entrenched. Matching newlines means that a pattern like
"[^"]*" can match the entire input unless there's another quote
in the input.
- A rule can have at most one instance of trailing context (the
โ/โ operator or the โ$โ operator). The start condition, โ^โ,
and "<<EOF>>" patterns can only occur at the beginning of a
pattern and, as well as with โ/โ and โ$โ, cannot be grouped
inside parentheses. A โ^โ which does not occur at the beginโ
ning of a rule or a โ$โ which does not occur at the end of a
rule loses its special properties and is treated as a normal
character.
- The following are illegal:
foo/bar$
<sc1>foo<sc2>bar
Note that the first of these, can be written "foo/bar\n".
- The following will result in โ$โ or โ^โ being treated as a norโ
mal character:
foo|(bar$)
foo|^bar
If what's wanted is a "foo" or a bar-followed-by-a-newline, the
following could be used (the special โ|โ action is explained
below):
foo |
bar$ /* action goes here */
A similar trick will work for matching a foo or a bar-at-the-
beginning-of-a-line.
๐๐๐ ๐๐๐ ๐๐๐๐๐ ๐๐ ๐๐๐๐๐๐๐
When the generated scanner is run, it analyzes its input looking
for strings which match any of its patterns. If it finds more than
one match, it takes the one matching the most text (for trailing
context rules, this includes the length of the trailing part, even
though it will then be returned to the input). If it finds two or
more matches of the same length, the rule listed first in the ๐ณ๐น๐ฒ๐
input file is chosen.
Once the match is determined, the text corresponding to the match
(called the tฬฒoฬฒkฬฒeฬฒnฬฒ) is made available in the global character
pointer yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ, and its length in the global integer yฬฒyฬฒlฬฒeฬฒnฬฒgฬฒ. The
aฬฒcฬฒtฬฒiฬฒoฬฒnฬฒ corresponding to the matched pattern is then executed (a
more detailed description of actions follows), and then the remainโ
ing input is scanned for another match.
If no match is found, then the default rule is executed: the next
character in the input is considered matched and copied to the
standard output. Thus, the simplest legal ๐ณ๐น๐ฒ๐
input is:
%%
which generates a scanner that simply copies its input (one
character at a time) to its output.
Note that yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ can be defined in two different ways: either as a
character pointer or as a character array. Which definition ๐ณ๐น๐ฒ๐
uses can be controlled by including one of the special directives
โ%pointerโ or โ%arrayโ in the first (definitions) section of flex
input. The default is โ%pointerโ, unless the -๐น ๐น๐ฒ๐
compatibility
option is used, in which case yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ will be an array. The advanโ
tage of using โ%pointerโ is substantially faster scanning and no
buffer overflow when matching very large tokens (unless not enough
dynamic memory is available). The disadvantage is that actions are
restricted in how they can modify yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ (see the next section),
and calls to the ๐๐ป๐ฝ๐๐() function destroy the present contents of
yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ, which can be a considerable porting headache when moving
between different ๐น๐ฒ๐
versions.
The advantage of โ%arrayโ is that yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ can be modified as much as
wanted, and calls to ๐๐ป๐ฝ๐๐() do not destroy yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ (see below).
Furthermore, existing ๐น๐ฒ๐
programs sometimes access yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ exterโ
nally using declarations of the form:
extern char yytext[];
This definition is erroneous when used with โ%pointerโ, but correct
for โ%arrayโ.
โ%arrayโ defines yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ to be an array of YYLMAX characters, which
defaults to a fairly large value. The size can be changed by simโ
ply #define'ing YYLMAX to a different value in the first section of
๐ณ๐น๐ฒ๐
input. As mentioned above, with โ%pointerโ yytext grows
dynamically to accommodate large tokens. While this means a
โ%pointerโ scanner can accommodate very large tokens (such as
matching entire blocks of comments), bear in mind that each time
the scanner must resize yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ it also must rescan the entire token
from the beginning, so matching such tokens can prove slow. yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ
presently does not dynamically grow if a call to ๐๐ป๐ฝ๐๐() results in
too much text being pushed back; instead, a run-time error results.
Also note that โ%arrayโ cannot be used with C++ scanner classes
(the c++ option; see below).
๐๐๐๐๐๐๐
Each pattern in a rule has a corresponding action, which can be any
arbitrary C statement. The pattern ends at the first non-escaped
whitespace character; the remainder of the line is its action. If
the action is empty, then when the pattern is matched the input
token is simply discarded. For example, here is the specification
for a program which deletes all occurrences of "zap me" from its
input:
%%
"zap me"
(It will copy all other characters in the input to the output since
they will be matched by the default rule.)
Here is a program which compresses multiple blanks and tabs down to
a single blank, and throws away whitespace found at the end of a
line:
%%
[ \t]+ putchar(' ');
[ \t]+$ /* ignore this token */
If the action contains a โ{โ, then the action spans till the balโ
ancing โ}โ is found, and the action may cross multiple lines. ๐ณ๐น๐ฒ๐
knows about C strings and comments and won't be fooled by braces
found within them, but also allows actions to begin with โ%{โ and
will consider the action to be all the text up to the next โ%}โ
(regardless of ordinary braces inside the action).
An action consisting solely of a vertical bar (โ|โ) means "same as
the action for the next rule". See below for an illustration.
Actions can include arbitrary C code, including return statements
to return a value to whatever routine called ๐๐๐น๐ฒ๐
(). Each time
๐๐๐น๐ฒ๐
() is called, it continues processing tokens from where it
last left off until it either reaches the end of the file or exeโ
cutes a return.
Actions are free to modify yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ except for lengthening it (adding
characters to its end - these will overwrite later characters in
the input stream). This, however, does not apply when using
โ%arrayโ (see above); in that case, yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ may be freely modified
in any way.
Actions are free to modify yฬฒyฬฒlฬฒeฬฒnฬฒgฬฒ except they should not do so if
the action also includes use of ๐๐๐บ๐ผ๐ฟ๐ฒ() (see below).
There are a number of special directives which can be included
within an action:
ECHO Copies yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ to the scanner's output.
BEGIN Followed by the name of a start condition, places the scanโ
ner in the corresponding start condition (see below).
REJECT Directs the scanner to proceed on to the "second best" rule
which matched the input (or a prefix of the input). The
rule is chosen as described above in HฬฒOฬฒWฬฒ TฬฒHฬฒEฬฒ IฬฒNฬฒPฬฒUฬฒTฬฒ IฬฒSฬฒ
MฬฒAฬฒTฬฒCฬฒHฬฒEฬฒDฬฒ, and yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ and yฬฒyฬฒlฬฒeฬฒnฬฒgฬฒ set up appropriately. It
may either be one which matched as much text as the origiโ
nally chosen rule but came later in the ๐ณ๐น๐ฒ๐
input file, or
one which matched less text. For example, the following
will both count the words in the input and call the routine
๐๐ฝ๐ฒ๐ฐ๐ถ๐ฎ๐น() whenever "frob" is seen:
int word_count = 0;
%%
frob special(); REJECT;
[^ \t\n]+ ++word_count;
Without the RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ, any "frob"'s in the input would not be
counted as words, since the scanner normally executes only
one action per token. Multiple RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ's are allowed, each
one finding the next best choice to the currently active
rule. For example, when the following scanner scans the
token "abcd", it will write "abcdabcaba" to the output:
%%
a |
ab |
abc |
abcd ECHO; REJECT;
.|\n /* eat up any unmatched character */
(The first three rules share the fourth's action since they
use the special โ|โ action.) RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ is a particularly
expensive feature in terms of scanner performance; if it is
used in any of the scanner's actions it will slow down all
of the scanner's matching. Furthermore, RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ cannot be
used with the -๐๐ณ or -๐๐
options (see below).
Note also that unlike the other special actions, RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ is
a bฬฒrฬฒaฬฒnฬฒcฬฒhฬฒ; code immediately following it in the action will
not be executed.
yymore()
Tells the scanner that the next time it matches a rule, the
corresponding token should be appended onto the current
value of yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ rather than replacing it. For example,
given the input "mega-kludge" the following will write
"mega-mega-kludge" to the output:
%%
mega- ECHO; yymore();
kludge ECHO;
First "mega-" is matched and echoed to the output. Then
"kludge" is matched, but the previous "mega-" is still
hanging around at the beginning of yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ so the EฬฒCฬฒHฬฒOฬฒ for
the "kludge" rule will actually write "mega-kludge".
Two notes regarding use of ๐๐๐บ๐ผ๐ฟ๐ฒ(): First, ๐๐๐บ๐ผ๐ฟ๐ฒ()
depends on the value of yฬฒyฬฒlฬฒeฬฒnฬฒgฬฒ correctly reflecting the
size of the current token, so yฬฒyฬฒlฬฒeฬฒnฬฒgฬฒ must not be modified
when using ๐๐๐บ๐ผ๐ฟ๐ฒ(). Second, the presence of ๐๐๐บ๐ผ๐ฟ๐ฒ() in
the scanner's action entails a minor performance penalty in
the scanner's matching speed.
yyless(n)
Returns all but the first nฬฒ characters of the current token
back to the input stream, where they will be rescanned when
the scanner looks for the next match. yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ and yฬฒyฬฒlฬฒeฬฒnฬฒgฬฒ
are adjusted appropriately (e.g., yฬฒyฬฒlฬฒeฬฒnฬฒgฬฒ will now be equal
to nฬฒ). For example, on the input "foobar" the following
will write out "foobarbar":
%%
foobar ECHO; yyless(3);
[a-z]+ ECHO;
An argument of 0 to yฬฒyฬฒlฬฒeฬฒsฬฒsฬฒ will cause the entire current
input string to be scanned again. Unless how the scanner
will subsequently process its input has been changed (using
BฬฒEฬฒGฬฒIฬฒNฬฒ, for example), this will result in an endless loop.
Note that yฬฒyฬฒlฬฒeฬฒsฬฒsฬฒ is a macro and can only be used in the
๐ณ๐น๐ฒ๐
input file, not from other source files.
unput(c)
Puts the character cฬฒ back into the input stream. It will
be the next character scanned. The following action will
take the current token and cause it to be rescanned
enclosed in parentheses.
{
int i;
char *yycopy;
/* Copy yytext because unput() trashes yytext */
if ((yycopy = strdup(yytext)) == NULL)
err(1, NULL);
unput(')');
for (i = yyleng - 1; i >= 0; --i)
unput(yycopy[i]);
unput('(');
free(yycopy);
}
Note that since each ๐๐ป๐ฝ๐๐() puts the given character back
at the beginning of the input stream, pushing back strings
must be done back-to-front.
An important potential problem when using ๐๐ป๐ฝ๐๐() is that
if using โ%pointerโ (the default), a call to ๐๐ป๐ฝ๐๐()
destroys the contents of yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ, starting with its rightโ
most character and devouring one character to the left with
each call. If the value of yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ should be preserved
after a call to ๐๐ป๐ฝ๐๐() (as in the above example), it must
either first be copied elsewhere, or the scanner must be
built using โ%arrayโ instead (see HฬฒOฬฒWฬฒ TฬฒHฬฒEฬฒ IฬฒNฬฒPฬฒUฬฒTฬฒ IฬฒSฬฒ
MฬฒAฬฒTฬฒCฬฒHฬฒEฬฒDฬฒ).
Finally, note that EOF cannot be put back to attempt to
mark the input stream with an end-of-file.
input()
Reads the next character from the input stream. For examโ
ple, the following is one way to eat up C comments:
%%
"/*" {
int c;
for (;;) {
while ((c = input()) != '*' && c != EOF)
; /* eat up text of comment */
if (c == '*') {
while ((c = input()) == '*')
;
if (c == '/')
break; /* found the end */
}
if (c == EOF) {
errx(1, "EOF in comment");
break;
}
}
}
(Note that if the scanner is compiled using C++, then
๐ถ๐ป๐ฝ๐๐() is instead referred to as ๐๐๐ถ๐ป๐ฝ๐๐(), in order to
avoid a name clash with the C++ stream by the name of
input.)
YY_FLUSH_BUFFER
Flushes the scanner's internal buffer so that the next time
the scanner attempts to match a token, it will first refill
the buffer using YY_INPUT (see TฬฒHฬฒEฬฒ GฬฒEฬฒNฬฒEฬฒRฬฒAฬฒTฬฒEฬฒDฬฒ SฬฒCฬฒAฬฒNฬฒNฬฒEฬฒRฬฒ,
below). This action is a special case of the more general
๐๐_๐ณ๐น๐๐๐ต_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ() function, described below in the section
MฬฒUฬฒLฬฒTฬฒIฬฒPฬฒLฬฒEฬฒ IฬฒNฬฒPฬฒUฬฒTฬฒ BฬฒUฬฒFฬฒFฬฒEฬฒRฬฒSฬฒ.
yyterminate()
Can be used in lieu of a return statement in an action. It
terminates the scanner and returns a 0 to the scanner's
caller, indicating "all done". By default, ๐๐๐๐ฒ๐ฟ๐บ๐ถ๐ป๐ฎ๐๐ฒ()
is also called when an end-of-file is encountered. It is a
macro and may be redefined.
๐๐๐ ๐๐๐๐๐๐๐๐๐ ๐๐๐๐๐๐๐
The output of ๐ณ๐น๐ฒ๐
is the file lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒ, which contains the scanโ
ning routine ๐๐๐น๐ฒ๐
(), a number of tables used by it for matching
tokens, and a number of auxiliary routines and macros. By default,
๐๐๐น๐ฒ๐
() is declared as follows:
int yylex()
{
... various definitions and the actions in here ...
}
(If the environment supports function prototypes, then it will be
"int yylex(void)".) This definition may be changed by defining the
YY_DECL macro. For example:
#define YY_DECL float lexscan(a, b) float a, b;
would give the scanning routine the name lฬฒeฬฒxฬฒsฬฒcฬฒaฬฒnฬฒ, returning a
float, and taking two floats as arguments. Note that if arguments
are given to the scanning routine using a K&R-style/non-prototyped
function declaration, the definition must be terminated with a
semi-colon (โ;โ).
Whenever ๐๐๐น๐ฒ๐
() is called, it scans tokens from the global input
file yฬฒyฬฒiฬฒnฬฒ (which defaults to stdin). It continues until it either
reaches an end-of-file (at which point it returns the value 0) or
one of its actions executes a rฬฒeฬฒtฬฒuฬฒrฬฒnฬฒ statement.
If the scanner reaches an end-of-file, subsequent calls are undeโ
fined unless either yฬฒyฬฒiฬฒnฬฒ is pointed at a new input file (in which
case scanning continues from that file), or ๐๐๐ฟ๐ฒ๐๐๐ฎ๐ฟ๐() is called.
๐๐๐ฟ๐ฒ๐๐๐ฎ๐ฟ๐() takes one argument, a FฬฒIฬฒLฬฒEฬฒ *ฬฒ pointer (which can be nil,
if YY_INPUT has been set up to scan from a source other than yฬฒyฬฒiฬฒnฬฒ),
and initializes yฬฒyฬฒiฬฒnฬฒ for scanning from that file. Essentially
there is no difference between just assigning yฬฒyฬฒiฬฒnฬฒ to a new input
file or using ๐๐๐ฟ๐ฒ๐๐๐ฎ๐ฟ๐() to do so; the latter is available for
compatibility with previous versions of ๐ณ๐น๐ฒ๐
, and because it can be
used to switch input files in the middle of scanning. It can also
be used to throw away the current input buffer, by calling it with
an argument of yฬฒyฬฒiฬฒnฬฒ; but better is to use YY_FLUSH_BUFFER (see
above). Note that ๐๐๐ฟ๐ฒ๐๐๐ฎ๐ฟ๐() does not reset the start condition
to IฬฒNฬฒIฬฒTฬฒIฬฒAฬฒLฬฒ (see SฬฒTฬฒAฬฒRฬฒTฬฒ CฬฒOฬฒNฬฒDฬฒIฬฒTฬฒIฬฒOฬฒNฬฒSฬฒ, below).
If ๐๐๐น๐ฒ๐
() stops scanning due to executing a rฬฒeฬฒtฬฒuฬฒrฬฒnฬฒ statement in
one of the actions, the scanner may then be called again and it
will resume scanning where it left off.
By default (and for purposes of efficiency), the scanner uses
block-reads rather than simple getc(3) calls to read characters
from yฬฒyฬฒiฬฒnฬฒ. The nature of how it gets its input can be controlled
by defining the YY_INPUT macro. YY_INPUT's calling sequence is
"YY_INPUT(buf,result,max_size)". Its action is to place up to
max_size characters in the character array bฬฒuฬฒfฬฒ and return in the
integer variable rฬฒeฬฒsฬฒuฬฒlฬฒtฬฒ either the number of characters read or the
constant YY_NULL (0 on UNIX systems) to indicate EOF. The default
YY_INPUT reads from the global file-pointer "yyin".
A sample definition of YY_INPUT (in the definitions section of the
input file):
%{
#define YY_INPUT(buf,result,max_size) \
{ \
int c = getchar(); \
result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
}
%}
This definition will change the input processing to occur one charโ
acter at a time.
When the scanner receives an end-of-file indication from YY_INPUT,
it then checks the ๐๐๐๐ฟ๐ฎ๐ฝ() function. If ๐๐๐๐ฟ๐ฎ๐ฝ() returns false
(zero), then it is assumed that the function has gone ahead and set
up yฬฒyฬฒiฬฒnฬฒ to point to another input file, and scanning continues. If
it returns true (non-zero), then the scanner terminates, returning
0 to its caller. Note that in either case, the start condition
remains unchanged; it does not revert to IฬฒNฬฒIฬฒTฬฒIฬฒAฬฒLฬฒ.
If you do not supply your own version of ๐๐๐๐ฟ๐ฎ๐ฝ(), then you must
either use โ%option noyywrapโ (in which case the scanner behaves as
though ๐๐๐๐ฟ๐ฎ๐ฝ() returned 1), or you must link with -๐น๐ณ๐น to obtain
the default version of the routine, which always returns 1.
Three routines are available for scanning from in-memory buffers
rather than files: ๐๐_๐๐ฐ๐ฎ๐ป_๐๐๐ฟ๐ถ๐ป๐ด(), ๐๐_๐๐ฐ๐ฎ๐ป_๐ฏ๐๐๐ฒ๐(), and
๐๐_๐๐ฐ๐ฎ๐ป_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ(). See the discussion of them below in the section
MฬฒUฬฒLฬฒTฬฒIฬฒPฬฒLฬฒEฬฒ IฬฒNฬฒPฬฒUฬฒTฬฒ BฬฒUฬฒFฬฒFฬฒEฬฒRฬฒSฬฒ.
The scanner writes its EฬฒCฬฒHฬฒOฬฒ output to the yฬฒyฬฒoฬฒuฬฒtฬฒ global (default,
stdout), which may be redefined by the user simply by assigning it
to some other FฬฒIฬฒLฬฒEฬฒ pointer.
๐๐๐๐๐ ๐๐๐๐๐๐๐๐๐๐
๐ณ๐น๐ฒ๐
provides a mechanism for conditionally activating rules. Any
rule whose pattern is prefixed with "โจscโฉ" will only be active when
the scanner is in the start condition named "sc". For example,
<STRING>[^"]* { /* eat up the string body ... */
...
}
will be active only when the scanner is in the "STRING" start conโ
dition, and
<INITIAL,STRING,QUOTE>\. { /* handle an escape ... */
...
}
will be active only when the current start condition is either
"INITIAL", "STRING", or "QUOTE".
Start conditions are declared in the definitions (first) section of
the input using unindented lines beginning with either โ%sโ or โ%xโ
followed by a list of names. The former declares iฬฒnฬฒcฬฒlฬฒuฬฒsฬฒiฬฒvฬฒeฬฒ start
conditions, the latter eฬฒxฬฒcฬฒlฬฒuฬฒsฬฒiฬฒvฬฒeฬฒ start conditions. A start condiโ
tion is activated using the BฬฒEฬฒGฬฒIฬฒNฬฒ action. Until the next BฬฒEฬฒGฬฒIฬฒNฬฒ
action is executed, rules with the given start condition will be
active and rules with other start conditions will be inactive. If
the start condition is inclusive, then rules with no start condiโ
tions at all will also be active. If it is exclusive, then only
rules qualified with the start condition will be active. A set of
rules contingent on the same exclusive start condition describe a
scanner which is independent of any of the other rules in the ๐ณ๐น๐ฒ๐
input. Because of this, exclusive start conditions make it easy to
specify "mini-scanners" which scan portions of the input that are
syntactically different from the rest (e.g., comments).
If the distinction between inclusive and exclusive start conditions
is still a little vague, here's a simple example illustrating the
connection between the two. The set of rules:
%s example
%%
<example>foo do_something();
bar something_else();
is equivalent to
%x example
%%
<example>foo do_something();
<INITIAL,example>bar something_else();
Without the โจINITIAL,exampleโฉ qualifier, the โbarโ pattern in the
second example wouldn't be active (i.e., couldn't match) when in
start condition โexampleโ. If we just used โจexampleโฉ to qualify
โbarโ, though, then it would only be active in โexampleโ and not in
IฬฒNฬฒIฬฒTฬฒIฬฒAฬฒLฬฒ, while in the first example it's active in both, because in
the first example the โexampleโ start condition is an inclusive
(โ%sโ) start condition.
Also note that the special start-condition specifier โโจ*โฉโ matches
every start condition. Thus, the above example could also have
been written:
%x example
%%
<example>foo do_something();
<*>bar something_else();
The default rule (to EฬฒCฬฒHฬฒOฬฒ any unmatched character) remains active
in start conditions. It is equivalent to:
<*>.|\n ECHO;
โBEGIN(0)โ returns to the original state where only the rules with
no start conditions are active. This state can also be referred to
as the start-condition IฬฒNฬฒIฬฒTฬฒIฬฒAฬฒLฬฒ, so โBEGIN(INITIAL)โ is equivalent
to โBEGIN(0)โ. (The parentheses around the start condition name
are not required but are considered good style.)
BฬฒEฬฒGฬฒIฬฒNฬฒ actions can also be given as indented code at the beginning
of the rules section. For example, the following will cause the
scanner to enter the "SPECIAL" start condition whenever ๐๐๐น๐ฒ๐
() is
called and the global variable eฬฒnฬฒtฬฒeฬฒrฬฒ_sฬฒpฬฒeฬฒcฬฒiฬฒaฬฒlฬฒ is true:
int enter_special;
%x SPECIAL
%%
if (enter_special)
BEGIN(SPECIAL);
<SPECIAL>blahblahblah
...more rules follow...
To illustrate the uses of start conditions, here is a scanner which
provides two different interpretations of a string like "123.456".
By default it will treat it as three tokens: the integer "123", a
dot (โ.โ), and the integer "456". But if the string is preceded
earlier in the line by the string "expect-floats" it will treat it
as a single token, the floating-point number 123.456:
%{
#include <math.h>
%}
%s expect
%%
expect-floats BEGIN(expect);
<expect>[0-9]+"."[0-9]+ {
printf("found a float, = %f\n",
atof(yytext));
}
<expect>\n {
/*
* That's the end of the line, so
* we need another "expect-number"
* before we'll recognize any more
* numbers.
*/
BEGIN(INITIAL);
}
[0-9]+ {
printf("found an integer, = %d\n",
atoi(yytext));
}
"." printf("found a dot\n");
Here is a scanner which recognizes (and discards) C comments while
maintaining a count of the current input line:
%x comment
%%
int line_num = 1;
"/*" BEGIN(comment);
<comment>[^*\n]* /* eat anything that's not a '*' */
<comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
<comment>\n ++line_num;
<comment>"*"+"/" BEGIN(INITIAL);
This scanner goes to a bit of trouble to match as much text as posโ
sible with each rule. In general, when attempting to write a high-
speed scanner try to match as much as possible in each rule, as
it's a big win.
Note that start-condition names are really integer values and can
be stored as such. Thus, the above could be extended in the folโ
lowing fashion:
%x comment foo
%%
int line_num = 1;
int comment_caller;
"/*" {
comment_caller = INITIAL;
BEGIN(comment);
}
...
<foo>"/*" {
comment_caller = foo;
BEGIN(comment);
}
<comment>[^*\n]* /* eat anything that's not a '*' */
<comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
<comment>\n ++line_num;
<comment>"*"+"/" BEGIN(comment_caller);
Furthermore, the current start condition can be accessed by using
the integer-valued YY_START macro. For example, the above assignโ
ments to cฬฒoฬฒmฬฒmฬฒeฬฒnฬฒtฬฒ_cฬฒaฬฒlฬฒlฬฒeฬฒrฬฒ could instead be written
comment_caller = YY_START;
Flex provides YYSTATE as an alias for YY_START (since that is
what's used by AT&T UNIX ๐น๐ฒ๐
).
Note that start conditions do not have their own name-space; %s's
and %x's declare names in the same fashion as #define's.
Finally, here's an example of how to match C-style quoted strings
using exclusive start conditions, including expanded escape
sequences (but not including checking for a string that's too
long):
%x str
%%
#define MAX_STR_CONST 1024
char string_buf[MAX_STR_CONST];
char *string_buf_ptr;
\" string_buf_ptr = string_buf; BEGIN(str);
<str>\" { /* saw closing quote - all done */
BEGIN(INITIAL);
*string_buf_ptr = '\0';
/*
* return string constant token type and
* value to parser
*/
}
<str>\n {
/* error - unterminated string constant */
/* generate error message */
}
<str>\\[0-7]{1,3} {
/* octal escape sequence */
int result;
(void) sscanf(yytext + 1, "%o", &result);
if (result > 0xff) {
/* error, constant is out-of-bounds */
} else
*string_buf_ptr++ = result;
}
<str>\\[0-9]+ {
/*
* generate error - bad escape sequence; something
* like '\48' or '\0777777'
*/
}
<str>\\n *string_buf_ptr++ = '\n';
<str>\\t *string_buf_ptr++ = '\t';
<str>\\r *string_buf_ptr++ = '\r';
<str>\\b *string_buf_ptr++ = '\b';
<str>\\f *string_buf_ptr++ = '\f';
<str>\\(.|\n) *string_buf_ptr++ = yytext[1];
<str>[^\\\n\"]+ {
char *yptr = yytext;
while (*yptr)
*string_buf_ptr++ = *yptr++;
}
Often, such as in some of the examples above, a whole bunch of
rules are all preceded by the same start condition(s). ๐ณ๐น๐ฒ๐
makes
this a little easier and cleaner by introducing a notion of start
condition sฬฒcฬฒoฬฒpฬฒeฬฒ. A start condition scope is begun with:
<SCs>{
where โSCsโ is a list of one or more start conditions. Inside the
start condition scope, every rule automatically has the prefix
โจSCsโฉ applied to it, until a โ}โ which matches the initial โ{โ.
So, for example,
<ESC>{
"\\n" return '\n';
"\\r" return '\r';
"\\f" return '\f';
"\\0" return '\0';
}
is equivalent to:
<ESC>"\\n" return '\n';
<ESC>"\\r" return '\r';
<ESC>"\\f" return '\f';
<ESC>"\\0" return '\0';
Start condition scopes may be nested.
Three routines are available for manipulating stacks of start conโ
ditions:
void yy_push_state(int new_state)
Pushes the current start condition onto the top of the
start condition stack and switches to nฬฒeฬฒwฬฒ_sฬฒtฬฒaฬฒtฬฒeฬฒ as though
โBEGIN new_stateโ had been used (recall that start
condition names are also integers).
void yy_pop_state()
Pops the top of the stack and switches to it via BฬฒEฬฒGฬฒIฬฒNฬฒ.
int yy_top_state()
Returns the top of the stack without altering the stack's
contents.
The start condition stack grows dynamically and so has no built-in
size limitation. If memory is exhausted, program execution aborts.
To use start condition stacks, scanners must include a โ%option
stackโ directive (see OฬฒPฬฒTฬฒIฬฒOฬฒNฬฒSฬฒ below).
๐๐๐๐๐๐๐๐ ๐๐๐๐๐ ๐๐๐
๐
๐๐๐
Some scanners (such as those which support "include" files) require
reading from several input streams. As ๐ณ๐น๐ฒ๐
scanners do a large
amount of buffering, one cannot control where the next input will
be read from by simply writing a YY_INPUT which is sensitive to the
scanning context. YY_INPUT is only called when the scanner reaches
the end of its buffer, which may be a long time after scanning a
statement such as an "include" which requires switching the input
source.
To negotiate these sorts of problems, ๐ณ๐น๐ฒ๐
provides a mechanism for
creating and switching between multiple input buffers. An input
buffer is created by using:
YY_BUFFER_STATE yy_create_buffer(FILE *file, int size)
which takes a FฬฒIฬฒLฬฒEฬฒ pointer and a sฬฒiฬฒzฬฒeฬฒ and creates a buffer associโ
ated with the given file and large enough to hold sฬฒiฬฒzฬฒeฬฒ characters
(when in doubt, use YY_BUF_SIZE for the size). It returns a
YY_BUFFER_STATE handle, which may then be passed to other routines
(see below). The YY_BUFFER_STATE type is a pointer to an opaque
โstruct yy_buffer_stateโ structure, so YY_BUFFER_STATE variables
may be safely initialized to โ((YY_BUFFER_STATE) 0)โ if desired,
and the opaque structure can also be referred to in order to corโ
rectly declare input buffers in source files other than that of
scanners. Note that the FฬฒIฬฒLฬฒEฬฒ pointer in the call to
๐๐_๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ() is only used as the value of yฬฒyฬฒiฬฒnฬฒ seen by
YY_INPUT; if YY_INPUT is redefined so that it no longer uses yฬฒyฬฒiฬฒnฬฒ,
then a nil FฬฒIฬฒLฬฒEฬฒ pointer can safely be passed to ๐๐_๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ().
To select a particular buffer to scan:
void yy_switch_to_buffer(YY_BUFFER_STATE new_buffer)
It switches the scanner's input buffer so subsequent tokens will
come from nฬฒeฬฒwฬฒ_bฬฒuฬฒfฬฒfฬฒeฬฒrฬฒ. Note that ๐๐_๐๐๐ถ๐๐ฐ๐ต_๐๐ผ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ() may be used
by ๐๐๐๐ฟ๐ฎ๐ฝ() to set things up for continued scanning, instead of
opening a new file and pointing yฬฒyฬฒiฬฒnฬฒ at it. Note also that switchโ
ing input sources via either ๐๐_๐๐๐ถ๐๐ฐ๐ต_๐๐ผ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ() or ๐๐๐๐ฟ๐ฎ๐ฝ() does
not change the start condition.
void yy_delete_buffer(YY_BUFFER_STATE buffer)
is used to reclaim the storage associated with a buffer. (bฬฒuฬฒfฬฒfฬฒeฬฒrฬฒ
can be nil, in which case the routine does nothing.) To clear the
current contents of a buffer:
void yy_flush_buffer(YY_BUFFER_STATE buffer)
This function discards the buffer's contents, so the next time the
scanner attempts to match a token from the buffer, it will first
fill the buffer anew using YY_INPUT.
๐๐_๐ป๐ฒ๐_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ() is an alias for ๐๐_๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ(), provided for
compatibility with the C++ use of nฬฒeฬฒwฬฒ and dฬฒeฬฒlฬฒeฬฒtฬฒeฬฒ for creating and
destroying dynamic objects.
Finally, the YY_CURRENT_BUFFER macro returns a YY_BUFFER_STATE hanโ
dle to the current buffer.
Here is an example of using these features for writing a scanner
which expands include files (the โจโจEOFโฉโฉ feature is discussed
below):
/*
* the "incl" state is used for picking up the name
* of an include file
*/
%x incl
%{
#define MAX_INCLUDE_DEPTH 10
YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
int include_stack_ptr = 0;
%}
%%
include BEGIN(incl);
[a-z]+ ECHO;
[^a-z\n]*\n? ECHO;
<incl>[ \t]* /* eat the whitespace */
<incl>[^ \t\n]+ { /* got the include file name */
if (include_stack_ptr >= MAX_INCLUDE_DEPTH)
errx(1, "Includes nested too deeply");
include_stack[include_stack_ptr++] =
YY_CURRENT_BUFFER;
yyin = fopen(yytext, "r");
if (yyin == NULL)
err(1, NULL);
yy_switch_to_buffer(
yy_create_buffer(yyin, YY_BUF_SIZE));
BEGIN(INITIAL);
}
<<EOF>> {
if (--include_stack_ptr < 0)
yyterminate();
else {
yy_delete_buffer(YY_CURRENT_BUFFER);
yy_switch_to_buffer(
include_stack[include_stack_ptr]);
}
}
Three routines are available for setting up input buffers for scanโ
ning in-memory strings instead of files. All of them create a new
input buffer for scanning the string, and return a corresponding
YY_BUFFER_STATE handle (which should be deleted afterwards using
๐๐_๐ฑ๐ฒ๐น๐ฒ๐๐ฒ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ()). They also switch to the new buffer using
๐๐_๐๐๐ถ๐๐ฐ๐ต_๐๐ผ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ(), so the next call to ๐๐๐น๐ฒ๐
() will start scanโ
ning the string.
yy_scan_string(const char *str)
Scans a NUL-terminated string.
yy_scan_bytes(const char *bytes, int len)
Scans lฬฒeฬฒnฬฒ bytes (including possibly NUL's) starting at
location bฬฒyฬฒtฬฒeฬฒsฬฒ.
Note that both of these functions create and scan a copy of the
string or bytes. (This may be desirable, since ๐๐๐น๐ฒ๐
() modifies
the contents of the buffer it is scanning.) The copy can be
avoided by using:
yy_scan_buffer(char *base, yy_size_t size)
Which scans the buffer starting at bฬฒaฬฒsฬฒeฬฒ, consisting of sฬฒiฬฒzฬฒeฬฒ
bytes, the last two bytes of which must be
YY_END_OF_BUFFER_CHAR (ASCII NUL). These last two bytes
are not scanned; thus, scanning consists of base[0] through
base[size-2], inclusive.
If bฬฒaฬฒsฬฒeฬฒ is not set up in this manner (i.e., forget the
final two YY_END_OF_BUFFER_CHAR bytes), then
๐๐_๐๐ฐ๐ฎ๐ป_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ() returns a nil pointer instead of creating
a new input buffer.
The type yฬฒyฬฒ_sฬฒiฬฒzฬฒeฬฒ_tฬฒ is an integral type which can be cast to
an integer expression reflecting the size of the buffer.
๐๐๐-๐๐
-๐
๐๐๐ ๐๐๐๐๐
The special rule "โจโจEOFโฉโฉ" indicates actions which are to be taken
when an end-of-file is encountered and ๐๐๐๐ฟ๐ฎ๐ฝ() returns non-zero
(i.e., indicates no further files to process). The action must
finish by doing one of four things:
- Assigning yฬฒyฬฒiฬฒnฬฒ to a new input file (in previous versions of
๐ณ๐น๐ฒ๐
, after doing the assignment, it was necessary to call the
special action YY_NEW_FILE; this is no longer necessary).
- Executing a rฬฒeฬฒtฬฒuฬฒrฬฒnฬฒ statement.
- Executing the special ๐๐๐๐ฒ๐ฟ๐บ๐ถ๐ป๐ฎ๐๐ฒ() action.
- Switching to a new buffer using ๐๐_๐๐๐ถ๐๐ฐ๐ต_๐๐ผ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ() as shown
in the example above.
โจโจEOFโฉโฉ rules may not be used with other patterns; they may only be
qualified with a list of start conditions. If an unqualified
โจโจEOFโฉโฉ rule is given, it applies to all start conditions which do
not already have โจโจEOFโฉโฉ actions. To specify an โจโจEOFโฉโฉ rule for
only the initial start condition, use
<INITIAL><<EOF>>
These rules are useful for catching things like unclosed comments.
An example:
%x quote
%%
...other rules for dealing with quotes...
<quote><<EOF>> {
error("unterminated quote");
yyterminate();
}
<<EOF>> {
if (*++filelist)
yyin = fopen(*filelist, "r");
else
yyterminate();
}
๐๐๐๐๐๐๐๐๐๐๐๐๐ ๐๐๐๐๐๐
The macro YY_USER_ACTION can be defined to provide an action which
is always executed prior to the matched rule's action. For examโ
ple, it could be #define'd to call a routine to convert yytext to
lower-case. When YY_USER_ACTION is invoked, the variable yฬฒyฬฒ_aฬฒcฬฒtฬฒ
gives the number of the matched rule (rules are numbered starting
with 1). For example, to profile how often each rule is matched,
the following would do the trick:
#define YY_USER_ACTION ++ctr[yy_act]
where cฬฒtฬฒrฬฒ is an array to hold the counts for the different rules.
Note that the macro YY_NUM_RULES gives the total number of rules
(including the default rule, even if -๐ is used), so a correct decโ
laration for cฬฒtฬฒrฬฒ is:
int ctr[YY_NUM_RULES];
The macro YY_USER_INIT may be defined to provide an action which is
always executed before the first scan (and before the scanner's
internal initializations are done). For example, it could be used
to call a routine to read in a data table or open a logging file.
The macro yy_set_interactive(is_interactive) can be used to control
whether the current buffer is considered iฬฒnฬฒtฬฒeฬฒrฬฒaฬฒcฬฒtฬฒiฬฒvฬฒeฬฒ. An interacโ
tive buffer is processed more slowly, but must be used when the
scanner's input source is indeed interactive to avoid problems due
to waiting to fill buffers (see the discussion of the -๐ flag
below). A non-zero value in the macro invocation marks the buffer
as interactive, a zero value as non-interactive. Note that use of
this macro overrides โ%option always-interactiveโ or โ%option
never-interactiveโ (see OฬฒPฬฒTฬฒIฬฒOฬฒNฬฒSฬฒ below). ๐๐_๐๐ฒ๐_๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐๐ฒ() must
be invoked prior to beginning to scan the buffer that is (or is
not) to be considered interactive.
The macro yy_set_bol(at_bol) can be used to control whether the
current buffer's scanning context for the next token match is done
as though at the beginning of a line. A non-zero macro argument
makes rules anchored with โ^โ active, while a zero argument makes
โ^โ rules inactive.
The macro YY_AT_BOL returns true if the next token scanned from the
current buffer will have โ^โ rules active, false otherwise.
In the generated scanner, the actions are all gathered in one large
switch statement and separated using YY_BREAK, which may be redeโ
fined. By default, it is simply a "break", to separate each rule's
action from the following rules. Redefining YY_BREAK allows, for
example, C++ users to โ#define YY_BREAKโ to do nothing (while being
very careful that every rule ends with a "break" or a "return"!)
to avoid suffering from unreachable statement warnings where
because a rule's action ends with โreturnโ, the YY_BREAK is inacโ
cessible.
๐๐๐๐๐๐ ๐๐๐๐๐๐๐๐๐ ๐๐ ๐๐๐ ๐๐๐๐
This section summarizes the various values available to the user in
the rule actions.
char *yytext
Holds the text of the current token. It may be modified
but not lengthened (characters cannot be appended to the
end).
If the special directive โ%arrayโ appears in the first secโ
tion of the scanner description, then yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ is instead
declared โchar yytext[YYLMAX]โ, where YYLMAX is a macro
definition that can be redefined in the first section to
change the default value (generally 8KB). Using โ%arrayโ
results in somewhat slower scanners, but the value of
yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ becomes immune to calls to ๐ถ๐ป๐ฝ๐๐() and ๐๐ป๐ฝ๐๐(),
which potentially destroy its value when yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ is a charโ
acter pointer. The opposite of โ%arrayโ is โ%pointerโ,
which is the default.
โ%arrayโ cannot be used when generating C++ scanner classes
(the -+ flag).
int yyleng
Holds the length of the current token.
FILE *yyin
Is the file which by default ๐ณ๐น๐ฒ๐
reads from. It may be
redefined, but doing so only makes sense before scanning
begins or after an EOF has been encountered. Changing it
in the midst of scanning will have unexpected results since
๐ณ๐น๐ฒ๐
buffers its input; use ๐๐๐ฟ๐ฒ๐๐๐ฎ๐ฟ๐() instead. Once
scanning terminates because an end-of-file has been seen,
yฬฒyฬฒiฬฒnฬฒ can be assigned as the new input file and the scanner
can be called again to continue scanning.
void yyrestart(FILE *new_file)
May be called to point yฬฒyฬฒiฬฒnฬฒ at the new input file. The
switch-over to the new file is immediate (any previously
buffered-up input is lost). Note that calling ๐๐๐ฟ๐ฒ๐๐๐ฎ๐ฟ๐()
with yฬฒyฬฒiฬฒnฬฒ as an argument thus throws away the current input
buffer and continues scanning the same input file.
FILE *yyout
Is the file to which EฬฒCฬฒHฬฒOฬฒ actions are done. It can be
reassigned by the user.
YY_CURRENT_BUFFER
Returns a YY_BUFFER_STATE handle to the current buffer.
YY_START
Returns an integer value corresponding to the current start
condition. This value can subsequently be used with BฬฒEฬฒGฬฒIฬฒNฬฒ
to return to that start condition.
๐๐๐๐๐๐
๐๐๐๐๐ ๐๐๐๐ ๐๐๐๐
One of the main uses of ๐ณ๐น๐ฒ๐
is as a companion to the yacc(1)
parser-generator. yacc parsers expect to call a routine named
๐๐๐น๐ฒ๐
() to find the next input token. The routine is supposed to
return the type of the next token as well as putting any associated
value in the global yฬฒyฬฒlฬฒvฬฒaฬฒlฬฒ, which is defined externally, and can be
a union or any other complex data structure. To use ๐ณ๐น๐ฒ๐
with
yacc, one specifies the -๐ฑ option to yacc to instruct it to generโ
ate the file yฬฒ.ฬฒtฬฒaฬฒbฬฒ.ฬฒhฬฒ containing definitions of all the โ%tokensโ
appearing in the yacc input. This file is then included in the
๐ณ๐น๐ฒ๐
scanner. For example, if one of the tokens is "TOK_NUMBER",
part of the scanner might look like:
%{
#include "y.tab.h"
%}
%%
[0-9]+ yylval = atoi(yytext); return TOK_NUMBER;
๐๐๐๐๐๐๐
๐ณ๐น๐ฒ๐
has the following options:
-๐ณ Instructs ๐ณ๐น๐ฒ๐
to generate a 7-bit scanner, i.e., one which
can only recognize 7-bit characters in its input. The
advantage of using -๐ณ is that the scanner's tables can be
up to half the size of those generated using the -๐ด option
(see below). The disadvantage is that such scanners often
hang or crash if their input contains an 8-bit character.
Note, however, that unless generating a scanner using the
-๐๐ณ or -๐๐
table compression options, use of -๐ณ will save
only a small amount of table space, and make the scanner
considerably less portable. ๐ณ๐น๐ฒ๐
's default behavior is to
generate an 8-bit scanner unless -๐๐ณ or -๐๐
is specified,
in which case ๐ณ๐น๐ฒ๐
defaults to generating 7-bit scanners
unless it was configured to generate 8-bit scanners (as
will often be the case with non-USA sites). It is possible
tell whether ๐ณ๐น๐ฒ๐
generated a 7-bit or an 8-bit scanner by
inspecting the flag summary in the -๐ output as described
below.
Note that if -๐๐ณ๐ฒ or -๐๐
๐ฒ are used (the table compression
options, but also using equivalence classes as discussed
below), ๐ณ๐น๐ฒ๐
still defaults to generating an 8-bit scanner,
since usually with these compression options full 8-bit
tables are not much more expensive than 7-bit tables.
-๐ด Instructs ๐ณ๐น๐ฒ๐
to generate an 8-bit scanner, i.e., one
which can recognize 8-bit characters. This flag is only
needed for scanners generated using -๐๐ณ or -๐๐
, as otherโ
wise ๐ณ๐น๐ฒ๐
defaults to generating an 8-bit scanner anyway.
See the discussion of -๐ณ above for ๐ณ๐น๐ฒ๐
's default behavior
and the tradeoffs between 7-bit and 8-bit scanners.
-๐ Instructs ๐ณ๐น๐ฒ๐
to generate a bฬฒaฬฒtฬฒcฬฒhฬฒ scanner, the opposite of
iฬฒnฬฒtฬฒeฬฒrฬฒaฬฒcฬฒtฬฒiฬฒvฬฒeฬฒ scanners generated by -๐ (see below). In genโ
eral, -๐ is used when the scanner will never be used interโ
actively, and you want to squeeze a little more performance
out of it. If the aim is instead to squeeze out a lot more
performance, use the -๐๐ณ or -๐๐
options (discussed below),
which turn on -๐ automatically anyway.
-๐ฏ Generate backing-up information to lฬฒeฬฒxฬฒ.ฬฒbฬฒaฬฒcฬฒkฬฒuฬฒpฬฒ. This is a
list of scanner states which require backing up and the
input characters on which they do so. By adding rules one
can remove backing-up states. If all backing-up states are
eliminated and -๐๐ณ or -๐๐
is used, the generated scanner
will run faster (see the -๐ฝ flag). Only users who wish to
squeeze every last cycle out of their scanners need worry
about this option. (See the section on PฬฒEฬฒRฬฒFฬฒOฬฒRฬฒMฬฒAฬฒNฬฒCฬฒEฬฒ
CฬฒOฬฒNฬฒSฬฒIฬฒDฬฒEฬฒRฬฒAฬฒTฬฒIฬฒOฬฒNฬฒSฬฒ below.)
-๐[๐ฎ๐ฒ๐
๐ณ๐บ๐ฟ]
Controls the degree of table compression and, more generโ
ally, trade-offs between small scanners and fast scanners.
-๐๐ฎ Instructs ๐ณ๐น๐ฒ๐
to trade off larger tables in the
generated scanner for faster performance because
the elements of the tables are better aligned for
memory access and computation. On some RISC archiโ
tectures, fetching and manipulating longwords is
more efficient than with smaller-sized units such
as shortwords. This option can double the size of
the tables used by the scanner.
-๐๐ฒ Directs ๐ณ๐น๐ฒ๐
to construct eฬฒqฬฒuฬฒiฬฒvฬฒaฬฒlฬฒeฬฒnฬฒcฬฒeฬฒ cฬฒlฬฒaฬฒsฬฒsฬฒeฬฒsฬฒ,
i.e., sets of characters which have identical lexiโ
cal properties (for example, if the only appearance
of digits in the ๐ณ๐น๐ฒ๐
input is in the character
class "[0-9]" then the digits โ0โ, โ1โ, โ...โ, โ9โ
will all be put in the same equivalence class).
Equivalence classes usually give dramatic reducโ
tions in the final table/object file sizes
(typically a factor of 2-5) and are pretty cheap
performance-wise (one array look-up per character
scanned).
-๐๐
Specifies that the alternate fast scanner represenโ
tation (described below under the -๐
option) should
be used. This option cannot be used with -+.
-๐๐ณ Specifies that the fฬฒuฬฒlฬฒlฬฒ scanner tables should be
generated - ๐ณ๐น๐ฒ๐
should not compress the tables by
taking advantage of similar transition functions
for different states.
-๐๐บ Directs ๐ณ๐น๐ฒ๐
to construct mฬฒeฬฒtฬฒaฬฒ-ฬฒeฬฒqฬฒuฬฒiฬฒvฬฒaฬฒlฬฒeฬฒnฬฒcฬฒeฬฒ cฬฒlฬฒaฬฒsฬฒsฬฒeฬฒsฬฒ,
which are sets of equivalence classes (or characโ
ters, if equivalence classes are not being used)
that are commonly used together. Meta-equivalence
classes are often a big win when using compressed
tables, but they have a moderate performance impact
(one or two "if" tests and one array look-up per
character scanned).
-๐๐ฟ Causes the generated scanner to bฬฒyฬฒpฬฒaฬฒsฬฒsฬฒ use of the
standard I/O library (stdio) for input. Instead of
calling fread(3) or getc(3), the scanner will use
the read(2) system call, resulting in a performance
gain which varies from system to system, but in
general is probably negligible unless -๐๐ณ or -๐๐
are being used. Using -๐๐ฟ can cause strange behavโ
ior if, for example, reading from yฬฒyฬฒiฬฒnฬฒ using stdio
prior to calling the scanner (because the scanner
will miss whatever text previous reads left in the
stdio input buffer).
-๐๐ฟ has no effect if YY_INPUT is defined (see TฬฒHฬฒEฬฒ
GฬฒEฬฒNฬฒEฬฒRฬฒAฬฒTฬฒEฬฒDฬฒ SฬฒCฬฒAฬฒNฬฒNฬฒEฬฒRฬฒ above).
A lone -๐ specifies that the scanner tables should be comโ
pressed but neither equivalence classes nor meta-equivaโ
lence classes should be used.
The options -๐๐ณ or -๐๐
and -๐๐บ do not make sense together -
there is no opportunity for meta-equivalence classes if the
table is not being compressed. Otherwise the options may
be freely mixed, and are cumulative.
The default setting is -๐๐ฒ๐บ which specifies that ๐ณ๐น๐ฒ๐
should generate equivalence classes and meta-equivalence
classes. This setting provides the highest degree of table
compression. It is possible to trade off faster-executing
scanners at the cost of larger tables with the following
generally being true:
slowest & smallest
-Cem
-Cm
-Ce
-C
-C{f,F}e
-C{f,F}
-C{f,F}a
fastest & largest
Note that scanners with the smallest tables are usually
generated and compiled the quickest, so during development
the default is usually best, maximal compression.
-๐๐ณ๐ฒ is often a good compromise between speed and size for
production scanners.
-๐ฑ Makes the generated scanner run in debug mode. Whenever a
pattern is recognized and the global yฬฒyฬฒ_fฬฒlฬฒeฬฒxฬฒ_dฬฒeฬฒbฬฒuฬฒgฬฒ is non-
zero (which is the default), the scanner will write to
stderr a line of the form:
--accepting rule at line 53 ("the matched text")
The line number refers to the location of the rule in the
file defining the scanner (i.e., the file that was fed to
๐ณ๐น๐ฒ๐
). Messages are also generated when the scanner backs
up, accepts the default rule, reaches the end of its input
buffer (or encounters a NUL; at this point, the two look
the same as far as the scanner's concerned), or reaches an
end-of-file.
-๐
Specifies that the fast scanner table representation should
be used (and stdio bypassed). This representation is about
as fast as the full table representation (-๐ณ), and for some
sets of patterns will be considerably smaller (and for
others, larger). In general, if the pattern set contains
both "keywords" and a catch-all, "identifier" rule, such as
in the set:
"case" return TOK_CASE;
"switch" return TOK_SWITCH;
...
"default" return TOK_DEFAULT;
[a-z]+ return TOK_ID;
then it's better to use the full table representation. If
only the "identifier" rule is present and a hash table or
some such is used to detect the keywords, it's better to
use -๐
.
This option is equivalent to -๐๐
๐ฟ (see above). It cannot
be used with -+.
-๐ณ Specifies fฬฒaฬฒsฬฒtฬฒ sฬฒcฬฒaฬฒnฬฒnฬฒeฬฒrฬฒ. No table compression is done and
stdio is bypassed. The result is large but fast. This
option is equivalent to -๐๐ณ๐ฟ (see above).
-๐ต Generates a help summary of ๐ณ๐น๐ฒ๐
's options to stdout and
then exits. -? and --๐ต๐ฒ๐น๐ฝ are synonyms for -๐ต.
-๐ Instructs ๐ณ๐น๐ฒ๐
to generate an iฬฒnฬฒtฬฒeฬฒrฬฒaฬฒcฬฒtฬฒiฬฒvฬฒeฬฒ scanner. An
interactive scanner is one that only looks ahead to decide
what token has been matched if it absolutely must. It
turns out that always looking one extra character ahead,
even if the scanner has already seen enough text to disamโ
biguate the current token, is a bit faster than only lookโ
ing ahead when necessary. But scanners that always look
ahead give dreadful interactive performance; for example,
when a user types a newline, it is not recognized as a newโ
line token until they enter aฬฒnฬฒoฬฒtฬฒhฬฒeฬฒrฬฒ token, which often
means typing in another whole line.
๐ณ๐น๐ฒ๐
scanners default to iฬฒnฬฒtฬฒeฬฒrฬฒaฬฒcฬฒtฬฒiฬฒvฬฒeฬฒ unless -๐๐ณ or -๐๐
taโ
ble-compression options are specified (see above). That's
because if high-performance is most important, one of these
options should be used, so if they weren't, ๐ณ๐น๐ฒ๐
assumes it
is preferable to trade off a bit of run-time performance
for intuitive interactive behavior. Note also that -๐ canโ
not be used in conjunction with -๐๐ณ or -๐๐
. Thus, this
option is not really needed; it is on by default for all
those cases in which it is allowed.
A scanner can be forced to not be interactive by using -๐
(see above).
-๐ถ Instructs ๐ณ๐น๐ฒ๐
to generate a case-insensitive scanner. The
case of letters given in the ๐ณ๐น๐ฒ๐
input patterns will be
ignored, and tokens in the input will be matched regardless
of case. The matched text given in yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ will have the
preserved case (i.e., it will not be folded).
-๐ Instructs ๐ณ๐น๐ฒ๐
not to generate โ#lineโ directives. Without
this option, ๐ณ๐น๐ฒ๐
peppers the generated scanner with #line
directives so error messages in the actions will be corโ
rectly located with respect to either the original ๐ณ๐น๐ฒ๐
input file (if the errors are due to code in the input
file), or lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒ (if the errors are ๐ณ๐น๐ฒ๐
's fault - these
sorts of errors should be reported to the email address
given below).
-๐น Turns on maximum compatibility with the original AT&T UNIX
๐น๐ฒ๐
implementation. Note that this does not mean full comโ
patibility. Use of this option costs a considerable amount
of performance, and it cannot be used with the -+, -๐ณ, -๐
,
-๐๐ณ, or -๐๐
options. For details on the compatibilities it
provides, see the section IฬฒNฬฒCฬฒOฬฒMฬฒPฬฒAฬฒTฬฒIฬฒBฬฒIฬฒLฬฒIฬฒTฬฒIฬฒEฬฒSฬฒ WฬฒIฬฒTฬฒHฬฒ LฬฒEฬฒXฬฒ AฬฒNฬฒDฬฒ
PฬฒOฬฒSฬฒIฬฒXฬฒ below. This option also results in the name
YY_FLEX_LEX_COMPAT being #define'd in the generated scanโ
ner.
-๐ป Another do-nothing, deprecated option included only for
POSIX compliance.
-๐ผoฬฒuฬฒtฬฒpฬฒuฬฒtฬฒ
Directs ๐ณ๐น๐ฒ๐
to write the scanner to the file oฬฒuฬฒtฬฒpฬฒuฬฒtฬฒ
instead of lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒ. If -๐ผ is combined with the -๐ option,
then the scanner is written to stdout but its โ#lineโ
directives (see the -๐ option above) refer to the file
oฬฒuฬฒtฬฒpฬฒuฬฒtฬฒ.
-๐pฬฒrฬฒeฬฒfฬฒiฬฒxฬฒ
Changes the default "yy" prefix used by ๐ณ๐น๐ฒ๐
for all globโ
ally visible variable and function names to instead be
pฬฒrฬฒeฬฒfฬฒiฬฒxฬฒ. For example, -๐fฬฒoฬฒoฬฒ changes the name of yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ to
fฬฒoฬฒoฬฒtฬฒeฬฒxฬฒtฬฒ. It also changes the name of the default output
file from lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒ to lฬฒeฬฒxฬฒ.ฬฒfฬฒoฬฒoฬฒ.ฬฒcฬฒ. Here are all of the names
affected:
yy_create_buffer
yy_delete_buffer
yy_flex_debug
yy_init_buffer
yy_flush_buffer
yy_load_buffer_state
yy_switch_to_buffer
yyin
yyleng
yylex
yylineno
yyout
yyrestart
yytext
yywrap
(If using a C++ scanner, then only yฬฒyฬฒwฬฒrฬฒaฬฒpฬฒ and yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ
are affected.) Within the scanner itself, it is still posโ
sible to refer to the global variables and functions using
either version of their name; but externally, they have the
modified name.
This option allows multiple ๐ณ๐น๐ฒ๐
programs to be easily
linked together into the same executable. Note, though,
that using this option also renames ๐๐๐๐ฟ๐ฎ๐ฝ(), so now either
an (appropriately named) version of the routine for the
scanner must be supplied, or โ%option noyywrapโ must be
used, as linking with -๐น๐ณ๐น no longer provides one by
default.
-๐ฝ Generates a performance report to stderr. The report conโ
sists of comments regarding features of the ๐ณ๐น๐ฒ๐
input file
which will cause a serious loss of performance in the
resulting scanner. If the flag is specified twice, comโ
ments regarding features that lead to minor performance
losses will also be reported>
Note that the use of RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ, โ%option yylinenoโ, and variโ
able trailing context (see the BฬฒUฬฒGฬฒSฬฒ section below) entails
a substantial performance penalty; use of ๐๐๐บ๐ผ๐ฟ๐ฒ(), the โ^โ
operator, and the -๐ flag entail minor performance penalโ
ties.
-๐sฬฒkฬฒeฬฒlฬฒeฬฒtฬฒoฬฒnฬฒ
Overrides the default skeleton file from which ๐ณ๐น๐ฒ๐
conโ
structs its scanners. This option is needed only for ๐ณ๐น๐ฒ๐
maintenance or development.
-๐ Causes the default rule (that unmatched scanner input is
echoed to stdout) to be suppressed. If the scanner encounโ
ters input that does not match any of its rules, it aborts
with an error. This option is useful for finding holes in
a scanner's rule set.
-๐ Makes ๐ณ๐น๐ฒ๐
run in tฬฒrฬฒaฬฒcฬฒeฬฒ mode. It will generate a lot of
messages to stderr concerning the form of the input and the
resultant non-deterministic and deterministic finite automโ
ata. This option is mostly for use in maintaining ๐ณ๐น๐ฒ๐
.
-๐ Instructs ๐ณ๐น๐ฒ๐
to write the scanner it generates to stanโ
dard output instead of lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒ.
-๐ Prints the version number to stdout and exits. --๐๐ฒ๐ฟ๐๐ถ๐ผ๐ป
is a synonym for -๐.
-๐ Specifies that ๐ณ๐น๐ฒ๐
should write to stderr a summary of
statistics regarding the scanner it generates. Most of the
statistics are meaningless to the casual ๐ณ๐น๐ฒ๐
user, but the
first line identifies the version of ๐ณ๐น๐ฒ๐
(same as reported
by -๐), and the next line the flags used when generating
the scanner, including those that are on by default.
-๐ Suppresses warning messages.
-+ Specifies that ๐ณ๐น๐ฒ๐
should generate a C++ scanner class.
See the section on GฬฒEฬฒNฬฒEฬฒRฬฒAฬฒTฬฒIฬฒNฬฒGฬฒ Cฬฒ+ฬฒ+ฬฒ SฬฒCฬฒAฬฒNฬฒNฬฒEฬฒRฬฒSฬฒ below for
details.
๐ณ๐น๐ฒ๐
also provides a mechanism for controlling options within the
scanner specification itself, rather than from the ๐ณ๐น๐ฒ๐
command
line. This is done by including โ%optionโ directives in the first
section of the scanner specification. Multiple options can be
specified with a single โ%optionโ directive, and multiple direcโ
tives in the first section of the ๐ณ๐น๐ฒ๐
input file.
Most options are given simply as names, optionally preceded by the
word "no" (with no intervening whitespace) to negate their meaning.
A number are equivalent to ๐ณ๐น๐ฒ๐
flags or their negation:
7bit -7 option
8bit -8 option
align -Ca option
backup -b option
batch -B option
c++ -+ option
caseful or
case-sensitive opposite of -i (default)
case-insensitive or
caseless -i option
debug -d option
default opposite of -s option
ecs -Ce option
fast -F option
full -f option
interactive -I option
lex-compat -l option
meta-ecs -Cm option
perf-report -p option
read -Cr option
stdout -t option
verbose -v option
warn opposite of -w option
(use "%option nowarn" for -w)
array equivalent to "%array"
pointer equivalent to "%pointer" (default)
Some %option's provide features otherwise not available:
always-interactive
Instructs ๐ณ๐น๐ฒ๐
to generate a scanner which always considers
its input "interactive". Normally, on each new input file
the scanner calls ๐ถ๐๐ฎ๐๐๐() in an attempt to determine
whether the scanner's input source is interactive and thus
should be read a character at a time. When this option is
used, however, no such call is made.
main Directs ๐ณ๐น๐ฒ๐
to provide a default ๐บ๐ฎ๐ถ๐ป() program for the
scanner, which simply calls ๐๐๐น๐ฒ๐
(). This option implies
โnoyywrapโ (see below).
never-interactive
Instructs ๐ณ๐น๐ฒ๐
to generate a scanner which never considers
its input "interactive" (again, no call made to ๐ถ๐๐ฎ๐๐๐()).
This is the opposite of โalways-interactiveโ.
stack Enables the use of start condition stacks (see SฬฒTฬฒAฬฒRฬฒTฬฒ
CฬฒOฬฒNฬฒDฬฒIฬฒTฬฒIฬฒOฬฒNฬฒSฬฒ above).
stdinit
If set (i.e., โ%option stdinitโ), initializes yฬฒyฬฒiฬฒnฬฒ and
yฬฒyฬฒoฬฒuฬฒtฬฒ to stdin and stdout, instead of the default of โnilโ.
Some existing ๐น๐ฒ๐
programs depend on this behavior, even
though it is not compliant with ANSI C, which does not
require stdin and stdout to be compile-time constant.
yylineno
Directs ๐ณ๐น๐ฒ๐
to generate a scanner that maintains the numโ
ber of the current line read from its input in the global
variable yฬฒyฬฒlฬฒiฬฒnฬฒeฬฒnฬฒoฬฒ. This option is implied by โ%option
lex-compatโ.
yywrap If unset (i.e., โ%option noyywrapโ), makes the scanner not
call ๐๐๐๐ฟ๐ฎ๐ฝ() upon an end-of-file, but simply assume that
there are no more files to scan (until the user points yฬฒyฬฒiฬฒnฬฒ
at a new file and calls ๐๐๐น๐ฒ๐
() again).
๐ณ๐น๐ฒ๐
scans rule actions to determine whether the RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ or ๐๐๐บ๐ผ๐ฟ๐ฒ()
features are being used. The โrejectโ and โyymoreโ options are
available to override its decision as to whether to use the
options, either by setting them (e.g., โ%option rejectโ) to indiโ
cate the feature is indeed used, or unsetting them to indicate it
actually is not used (e.g., โ%option noyymoreโ).
Three options take string-delimited values, offset with โ=โ:
%option outfile="ABC"
is equivalent to -๐ผAฬฒBฬฒCฬฒ, and
%option prefix="XYZ"
is equivalent to -๐XฬฒYฬฒZฬฒ. Finally,
%option yyclass="foo"
only applies when generating a C++ scanner (-+ option). It informs
๐ณ๐น๐ฒ๐
that โfooโ has been derived as a subclass of yyFlexLexer, so
๐ณ๐น๐ฒ๐
will place actions in the member function โfoo::yylex()โ
instead of โyyFlexLexer::yylex()โ. It also generates a
โyyFlexLexer::yylex()โ member function that emits a run-time error
(by invoking โyyFlexLexer::LexerError()โ) if called. See
GฬฒEฬฒNฬฒEฬฒRฬฒAฬฒTฬฒIฬฒNฬฒGฬฒ Cฬฒ+ฬฒ+ฬฒ SฬฒCฬฒAฬฒNฬฒNฬฒEฬฒRฬฒSฬฒ, below, for additional information.
A number of options are available for lint purists who want to supโ
press the appearance of unneeded routines in the generated scanner.
Each of the following, if unset (e.g., โ%option nounputโ), results
in the corresponding routine not appearing in the generated scanโ
ner:
input, unput
yy_push_state, yy_pop_state, yy_top_state
yy_scan_buffer, yy_scan_bytes, yy_scan_string
(though ๐๐_๐ฝ๐๐๐ต_๐๐๐ฎ๐๐ฒ() and friends won't appear anyway unless
โ%option stackโ is being used).
๐๐๐๐
๐๐๐๐๐๐๐ ๐๐๐๐๐๐๐๐๐๐๐๐๐๐
The main design goal of ๐ณ๐น๐ฒ๐
is that it generate high-performance
scanners. It has been optimized for dealing well with large sets
of rules. Aside from the effects on scanner speed of the table
compression -๐ options outlined above, there are a number of
options/actions which degrade performance. These are, from most
expensive to least:
REJECT
%option yylineno
arbitrary trailing context
pattern sets that require backing up
%array
%option interactive
%option always-interactive
'^' beginning-of-line operator
yymore()
with the first three all being quite expensive and the last two
being quite cheap. Note also that ๐๐ป๐ฝ๐๐() is implemented as a rouโ
tine call that potentially does quite a bit of work, while ๐๐๐น๐ฒ๐๐()
is a quite-cheap macro; so if just putting back some excess text,
use ๐๐๐น๐ฒ๐๐().
RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ should be avoided at all costs when performance is imporโ
tant. It is a particularly expensive option.
Getting rid of backing up is messy and often may be an enormous
amount of work for a complicated scanner. In principal, one begins
by using the -๐ฏ flag to generate a lฬฒeฬฒxฬฒ.ฬฒbฬฒaฬฒcฬฒkฬฒuฬฒpฬฒ file. For example,
on the input
%%
foo return TOK_KEYWORD;
foobar return TOK_KEYWORD;
the file looks like:
State #6 is non-accepting -
associated rule line numbers:
2 3
out-transitions: [ o ]
jam-transitions: EOF [ \001-n p-\177 ]
State #8 is non-accepting -
associated rule line numbers:
3
out-transitions: [ a ]
jam-transitions: EOF [ \001-` b-\177 ]
State #9 is non-accepting -
associated rule line numbers:
3
out-transitions: [ r ]
jam-transitions: EOF [ \001-q s-\177 ]
Compressed tables always back up.
The first few lines tell us that there's a scanner state in which
it can make a transition on an โoโ but not on any other character,
and that in that state the currently scanned text does not match
any rule. The state occurs when trying to match the rules found at
lines 2 and 3 in the input file. If the scanner is in that state
and then reads something other than an โoโ, it will have to back up
to find a rule which is matched. With a bit of headscratching one
can see that this must be the state it's in when it has seen โfoโ.
When this has happened, if anything other than another โoโ is seen,
the scanner will have to back up to simply match the โfโ (by the
default rule).
The comment regarding State #8 indicates there's a problem when
"foob" has been scanned. Indeed, on any character other than an
โaโ, the scanner will have to back up to accept "foo". Similarly,
the comment for State #9 concerns when "fooba" has been scanned and
an โrโ does not follow.
The final comment reminds us that there's no point going to all the
trouble of removing backing up from the rules unless we're using
-๐๐ณ or -๐๐
, since there's no performance gain doing so with comโ
pressed scanners.
The way to remove the backing up is to add "error" rules:
%%
foo return TOK_KEYWORD;
foobar return TOK_KEYWORD;
fooba |
foob |
fo {
/* false alarm, not really a keyword */
return TOK_ID;
}
Eliminating backing up among a list of keywords can also be done
using a "catch-all" rule:
%%
foo return TOK_KEYWORD;
foobar return TOK_KEYWORD;
[a-z]+ return TOK_ID;
This is usually the best solution when appropriate.
Backing up messages tend to cascade. With a complicated set of
rules it's not uncommon to get hundreds of messages. If one can
decipher them, though, it often only takes a dozen or so rules to
eliminate the backing up (though it's easy to make a mistake and
have an error rule accidentally match a valid token; a possible
future ๐ณ๐น๐ฒ๐
feature will be to automatically add rules to eliminate
backing up).
It's important to keep in mind that the benefits of eliminating
backing up are gained only if eฬฒvฬฒeฬฒrฬฒyฬฒ instance of backing up is elimโ
inated. Leaving just one gains nothing.
Vฬฒaฬฒrฬฒiฬฒaฬฒbฬฒlฬฒeฬฒ trailing context (where both the leading and trailing
parts do not have a fixed length) entails almost the same perforโ
mance loss as RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ (i.e., substantial). So when possible a rule
like:
%%
mouse|rat/(cat|dog) run();
is better written:
%%
mouse/cat|dog run();
rat/cat|dog run();
or as
%%
mouse|rat/cat run();
mouse|rat/dog run();
Note that here the special โ|โ action does not provide any savings,
and can even make things worse (see BฬฒUฬฒGฬฒSฬฒ below).
Another area where the user can increase a scanner's performance
(and one that's easier to implement) arises from the fact that the
longer the tokens matched, the faster the scanner will run. This
is because with long tokens the processing of most input characters
takes place in the (short) inner scanning loop, and does not often
have to go through the additional work of setting up the scanning
environment (e.g., yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ) for the action. Recall the scanner for
C comments:
%x comment
%%
int line_num = 1;
"/*" BEGIN(comment);
<comment>[^*\n]*
<comment>"*"+[^*/\n]*
<comment>\n ++line_num;
<comment>"*"+"/" BEGIN(INITIAL);
This could be sped up by writing it as:
%x comment
%%
int line_num = 1;
"/*" BEGIN(comment);
<comment>[^*\n]*
<comment>[^*\n]*\n ++line_num;
<comment>"*"+[^*/\n]*
<comment>"*"+[^*/\n]*\n ++line_num;
<comment>"*"+"/" BEGIN(INITIAL);
Now instead of each newline requiring the processing of another
action, recognizing the newlines is "distributed" over the other
rules to keep the matched text as long as possible. Note that
adding rules does nฬฒoฬฒtฬฒ slow down the scanner! The speed of the
scanner is independent of the number of rules or (modulo the conโ
siderations given at the beginning of this section) how complicated
the rules are with regard to operators such as โ*โ and โ|โ.
A final example in speeding up a scanner: scan through a file conโ
taining identifiers and keywords, one per line and with no other
extraneous characters, and recognize all the keywords. A natural
first approach is:
%%
asm |
auto |
break |
... etc ...
volatile |
while /* it's a keyword */
.|\n /* it's not a keyword */
To eliminate the back-tracking, introduce a catch-all rule:
%%
asm |
auto |
break |
... etc ...
volatile |
while /* it's a keyword */
[a-z]+ |
.|\n /* it's not a keyword */
Now, if it's guaranteed that there's exactly one word per line,
then we can reduce the total number of matches by a half by merging
in the recognition of newlines with that of the other tokens:
%%
asm\n |
auto\n |
break\n |
... etc ...
volatile\n |
while\n /* it's a keyword */
[a-z]+\n |
.|\n /* it's not a keyword */
One has to be careful here, as we have now reintroduced backing up
into the scanner. In particular, while we know that there will
never be any characters in the input stream other than letters or
newlines, ๐ณ๐น๐ฒ๐
can't figure this out, and it will plan for possibly
needing to back up when it has scanned a token like "auto" and then
the next character is something other than a newline or a letter.
Previously it would then just match the "auto" rule and be done,
but now it has no "auto" rule, only an "auto\n" rule. To eliminate
the possibility of backing up, we could either duplicate all rules
but without final newlines or, since we never expect to encounter
such an input and therefore don't how it's classified, we can
introduce one more catch-all rule, this one which doesn't include a
newline:
%%
asm\n |
auto\n |
break\n |
... etc ...
volatile\n |
while\n /* it's a keyword */
[a-z]+\n |
[a-z]+ |
.|\n /* it's not a keyword */
Compiled with -๐๐ณ, this is about as fast as one can get a ๐ณ๐น๐ฒ๐
scanner to go for this particular problem.
A final note: ๐ณ๐น๐ฒ๐
is slow when matching NUL's, particularly when a
token contains multiple NUL's. It's best to write rules which
match short amounts of text if it's anticipated that the text will
often include NUL's.
Another final note regarding performance: as mentioned above in the
section HฬฒOฬฒWฬฒ TฬฒHฬฒEฬฒ IฬฒNฬฒPฬฒUฬฒTฬฒ IฬฒSฬฒ MฬฒAฬฒTฬฒCฬฒHฬฒEฬฒDฬฒ, dynamically resizing yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ to
accommodate huge tokens is a slow process because it presently
requires that the (huge) token be rescanned from the beginning.
Thus if performance is vital, it is better to attempt to match
"large" quantities of text but not "huge" quantities, where the
cutoff between the two is at about 8K characters/token.
๐๐๐๐๐๐๐๐๐๐ ๐++ ๐๐๐๐๐๐๐๐
๐ณ๐น๐ฒ๐
provides two different ways to generate scanners for use with
C++. The first way is to simply compile a scanner generated by
๐ณ๐น๐ฒ๐
using a C++ compiler instead of a C compiler. This should not
generate any compilation errors (please report any found to the
email address given in the AฬฒUฬฒTฬฒHฬฒOฬฒRฬฒSฬฒ section below). C++ code can
then be used in rule actions instead of C code. Note that the
default input source for scanners remains yฬฒyฬฒiฬฒnฬฒ, and default echoing
is still done to yฬฒyฬฒoฬฒuฬฒtฬฒ. Both of these remain FฬฒIฬฒLฬฒEฬฒ *ฬฒ variables and
not C++ streams.
๐ณ๐น๐ฒ๐
can also be used to generate a C++ scanner class, using the -+
option (or, equivalently, โ%option c++โ), which is automatically
specified if the name of the flex executable ends in a โ+โ, such as
๐ณ๐น๐ฒ๐
++. When using this option, ๐ณ๐น๐ฒ๐
defaults to generating the
scanner to the file lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒcฬฒ instead of lฬฒeฬฒxฬฒ.ฬฒyฬฒyฬฒ.ฬฒcฬฒ. The generated
scanner includes the header file <gฬฒ+ฬฒ+ฬฒ/ฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ.ฬฒhฬฒ>, which defines
the interface to two C++ classes.
The first class, FฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ, provides an abstract base class definโ
ing the general scanner class interface. It provides the following
member functions:
const char* YYText()
Returns the text of the most recently matched token, the
equivalent of yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ.
int YYLeng()
Returns the length of the most recently matched token, the
equivalent of yฬฒyฬฒlฬฒeฬฒnฬฒgฬฒ.
int lineno() const
Returns the current input line number (see โ%option
yylinenoโ), or 1 if โ%option yylinenoโ was not used.
void set_debug(int flag)
Sets the debugging flag for the scanner, equivalent to
assigning to yฬฒyฬฒ_fฬฒlฬฒeฬฒxฬฒ_dฬฒeฬฒbฬฒuฬฒgฬฒ (see the OฬฒPฬฒTฬฒIฬฒOฬฒNฬฒSฬฒ section above).
Note that the scanner must be built using โ%option debugโ
to include debugging information in it.
int debug() const
Returns the current setting of the debugging flag.
Also provided are member functions equivalent to
๐๐_๐๐๐ถ๐๐ฐ๐ต_๐๐ผ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ(), ๐๐_๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ() (though the first arguโ
ment is an sฬฒtฬฒdฬฒ:ฬฒ:ฬฒiฬฒsฬฒtฬฒrฬฒeฬฒaฬฒmฬฒ*ฬฒ object pointer and not a FฬฒIฬฒLฬฒEฬฒ*ฬฒ),
๐๐_๐ณ๐น๐๐๐ต_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ(), ๐๐_๐ฑ๐ฒ๐น๐ฒ๐๐ฒ_๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ(), and ๐๐๐ฟ๐ฒ๐๐๐ฎ๐ฟ๐() (again, the
first argument is an sฬฒtฬฒdฬฒ:ฬฒ:ฬฒiฬฒsฬฒtฬฒrฬฒeฬฒaฬฒmฬฒ*ฬฒ object pointer).
The second class defined in <gฬฒ+ฬฒ+ฬฒ/ฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ.ฬฒhฬฒ> is yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ, which
is derived from FฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ. It defines the following additional
member functions:
yyFlexLexer(std::istream* arg_yyin = 0, std::ostream* arg_yyout =
0)
Constructs a yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ object using the given streams for
input and output. If not specified, the streams default to
cฬฒiฬฒnฬฒ and cฬฒoฬฒuฬฒtฬฒ, respectively.
virtual int yylex()
Performs the same role as ๐๐๐น๐ฒ๐
() does for ordinary flex
scanners: it scans the input stream, consuming tokens,
until a rule's action returns a value. If subclass โSโ is
derived from yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ, in order to access the member
functions and variables of โSโ inside ๐๐๐น๐ฒ๐
(), use โ%option
yyclass="S"โ to inform ๐ณ๐น๐ฒ๐
that the โSโ subclass will be
used instead of yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ. In this case, rather than
generating โyyFlexLexer::yylex()โ, ๐ณ๐น๐ฒ๐
generates
โS::yylex()โ (and also generates a dummy
โyyFlexLexer::yylex()โ that calls
โyyFlexLexer::LexerError()โ if called).
virtual void switch_streams(std::istream* new_in = 0, std::ostream*
new_out = 0)
Reassigns yฬฒyฬฒiฬฒnฬฒ to nฬฒeฬฒwฬฒ_iฬฒnฬฒ (if non-nil) and yฬฒyฬฒoฬฒuฬฒtฬฒ to nฬฒeฬฒwฬฒ_oฬฒuฬฒtฬฒ
(ditto), deleting the previous input buffer if yฬฒyฬฒiฬฒnฬฒ is
reassigned.
int yylex(std::istream* new_in, std::ostream* new_out = 0)
First switches the input streams via
โswitch_streams(new_in, new_out)โ and then returns the
value of ๐๐๐น๐ฒ๐
().
In addition, yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ defines the following protected virtual
functions which can be redefined in derived classes to tailor the
scanner:
virtual int LexerInput(char* buf, int max_size)
Reads up to mฬฒaฬฒxฬฒ_sฬฒiฬฒzฬฒeฬฒ characters into bฬฒuฬฒfฬฒ and returns the
number of characters read. To indicate end-of-input,
return 0 characters. Note that "interactive" scanners (see
the -๐ and -๐ flags) define the macro YY_INTERACTIVE. If
๐๐ฒ๐
๐ฒ๐ฟ๐๐ป๐ฝ๐๐() has been redefined, and it's necessary to take
different actions depending on whether or not the scanner
might be scanning an interactive input source, it's possiโ
ble to test for the presence of this name via โ#ifdefโ.
virtual void LexerOutput(const char* buf, int size)
Writes out sฬฒiฬฒzฬฒeฬฒ characters from the buffer bฬฒuฬฒfฬฒ, which,
while NUL-terminated, may also contain "internal" NUL's if
the scanner's rules can match text with NUL's in them.
virtual void LexerError(const char* msg)
Reports a fatal error message. The default version of this
function writes the message to the stream cฬฒeฬฒrฬฒrฬฒ and exits.
Note that a yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ object contains its entire scanning state.
Thus such objects can be used to create reentrant scanners. Multiโ
ple instances of the same yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ class can be instantiated,
and multiple C++ scanner classes can be combined in the same proโ
gram using the -๐ option discussed above.
Finally, note that the โ%arrayโ feature is not available to C++
scanner classes; โ%pointerโ must be used (the default).
Here is an example of a simple C++ scanner:
// An example of using the flex C++ scanner class.
%{
#include <errno.h>
int mylineno = 0;
%}
string \"[^\n"]+\"
ws [ \t]+
alpha [A-Za-z]
dig [0-9]
name ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])*
num1 [-+]?{dig}+\.?([eE][-+]?{dig}+)?
num2 [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)?
number {num1}|{num2}
%%
{ws} /* skip blanks and tabs */
"/*" {
int c;
while ((c = yyinput()) != 0) {
if(c == '\n')
++mylineno;
else if(c == '*') {
if ((c = yyinput()) == '/')
break;
else
unput(c);
}
}
}
{number} cout << "number " << YYText() << '\n';
\n mylineno++;
{name} cout << "name " << YYText() << '\n';
{string} cout << "string " << YYText() << '\n';
%%
int main(int /* argc */, char** /* argv */)
{
FlexLexer* lexer = new yyFlexLexer;
while(lexer->yylex() != 0)
;
return 0;
}
To create multiple (different) lexer classes, use the -๐ flag (or
the โprefix=โ option) to rename each yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ to some other
xฬฒxฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ. <gฬฒ+ฬฒ+ฬฒ/ฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ.ฬฒhฬฒ> can then be included in other
sources once per lexer class, first renaming yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ as folโ
lows:
#undef yyFlexLexer
#define yyFlexLexer xxFlexLexer
#include <g++/FlexLexer.h>
#undef yyFlexLexer
#define yyFlexLexer zzFlexLexer
#include <g++/FlexLexer.h>
If, for example, โ%option prefix="xx"โ is used for one scanner and
โ%option prefix="zz"โ is used for the other.
๐๐๐๐๐๐๐๐๐: the present form of the scanning class is experimental
and may change considerably between major releases.
๐๐๐๐๐๐๐๐๐๐๐๐๐๐๐๐๐ ๐๐๐๐ ๐๐๐ ๐๐๐ ๐๐๐๐๐
๐ณ๐น๐ฒ๐
is a rewrite of the AT&T UNIX ๐น๐ฒ๐
tool (the two implementaโ
tions do not share any code, though), with some extensions and
incompatibilities, both of which are of concern to those who wish
to write scanners acceptable to either implementation. ๐ณ๐น๐ฒ๐
is
fully compliant with the POSIX ๐น๐ฒ๐
specification, except that when
using โ%pointerโ (the default), a call to ๐๐ป๐ฝ๐๐() destroys the conโ
tents of yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ, which is counter to the POSIX specification.
In this section we discuss all of the known areas of incompatibilโ
ity between ๐ณ๐น๐ฒ๐
, AT&T UNIX ๐น๐ฒ๐
, and the POSIX specification.
๐ณ๐น๐ฒ๐
's -๐น option turns on maximum compatibility with the original
AT&T UNIX ๐น๐ฒ๐
implementation, at the cost of a major loss in the
generated scanner's performance. We note below which incompatibilโ
ities can be overcome using the -๐น option.
๐ณ๐น๐ฒ๐
is fully compatible with ๐น๐ฒ๐
with the following exceptions:
- The undocumented ๐น๐ฒ๐
scanner internal variable yฬฒyฬฒlฬฒiฬฒnฬฒeฬฒnฬฒoฬฒ is not
supported unless -๐น or โ%option yylinenoโ is used.
yฬฒyฬฒlฬฒiฬฒnฬฒeฬฒnฬฒoฬฒ should be maintained on a per-buffer basis, rather
than a per-scanner (single global variable) basis.
yฬฒyฬฒlฬฒiฬฒnฬฒeฬฒnฬฒoฬฒ is not part of the POSIX specification.
- The ๐ถ๐ป๐ฝ๐๐() routine is not redefinable, though it may be called
to read characters following whatever has been matched by a
rule. If ๐ถ๐ป๐ฝ๐๐() encounters an end-of-file, the normal
๐๐๐๐ฟ๐ฎ๐ฝ() processing is done. A โrealโ end-of-file is returned
by ๐ถ๐ป๐ฝ๐๐() as EOF.
Input is instead controlled by defining the YY_INPUT macro.
The ๐ณ๐น๐ฒ๐
restriction that ๐ถ๐ป๐ฝ๐๐() cannot be redefined is in
accordance with the POSIX specification, which simply does not
specify any way of controlling the scanner's input other than
by making an initial assignment to yฬฒyฬฒiฬฒnฬฒ.
- The ๐๐ป๐ฝ๐๐() routine is not redefinable. This restriction is in
accordance with POSIX.
- ๐ณ๐น๐ฒ๐
scanners are not as reentrant as ๐น๐ฒ๐
scanners. In particโ
ular, if a scanner is interactive and an interrupt handler
long-jumps out of the scanner, and the scanner is subsequently
called again, the following error message may be displayed:
fatal flex scanner internal error--end of buffer missed
To reenter the scanner, first use
yyrestart(yyin);
Note that this call will throw away any buffered input; usually
this isn't a problem with an interactive scanner.
Also note that flex C++ scanner classes are reentrant, so if
using C++ is an option , they should be used instead. See
GฬฒEฬฒNฬฒEฬฒRฬฒAฬฒTฬฒIฬฒNฬฒGฬฒ Cฬฒ+ฬฒ+ฬฒ SฬฒCฬฒAฬฒNฬฒNฬฒEฬฒRฬฒSฬฒ above for details.
- ๐ผ๐๐๐ฝ๐๐() is not supported. Output from the EฬฒCฬฒHฬฒOฬฒ macro is done
to the file-pointer yฬฒyฬฒoฬฒuฬฒtฬฒ (default stdout).
๐ผ๐๐๐ฝ๐๐() is not part of the POSIX specification.
- ๐น๐ฒ๐
does not support exclusive start conditions (%x), though
they are in the POSIX specification.
- When definitions are expanded, ๐ณ๐น๐ฒ๐
encloses them in parentheโ
ses. With ๐น๐ฒ๐
, the following:
NAME [A-Z][A-Z0-9]*
%%
foo{NAME}? printf("Found it\n");
%%
will not match the string "foo" because when the macro is
expanded the rule is equivalent to "foo[A-Z][A-Z0-9]*?" and the
precedence is such that the โ?โ is associated with "[A-Z0-9]*".
With ๐ณ๐น๐ฒ๐
, the rule will be expanded to "foo([A-Z][A-Z0-9]*)?"
and so the string "foo" will match.
Note that if the definition begins with โ^โ or ends with โ$โ
then it is not expanded with parentheses, to allow these operaโ
tors to appear in definitions without losing their special
meanings. But the โโจsโฉโ, โ/โ, and โจโจEOFโฉโฉ operators cannot be
used in a ๐ณ๐น๐ฒ๐
definition.
Using -๐น results in the ๐น๐ฒ๐
behavior of no parentheses around
the definition.
The POSIX specification is that the definition be enclosed in
parentheses.
- Some implementations of ๐น๐ฒ๐
allow a rule's action to begin on a
separate line, if the rule's pattern has trailing whitespace:
%%
foo|bar<space here>
{ foobar_action(); }
๐ณ๐น๐ฒ๐
does not support this feature.
- The ๐น๐ฒ๐
โ%rโ (generate a Ratfor scanner) option is not supโ
ported. It is not part of the POSIX specification.
- After a call to ๐๐ป๐ฝ๐๐(), yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ is undefined until the next
token is matched, unless the scanner was built using โ%arrayโ.
This is not the case with ๐น๐ฒ๐
or the POSIX specification. The
-๐น option does away with this incompatibility.
- The precedence of the โ{}โ (numeric range) operator is differโ
ent. ๐น๐ฒ๐
interprets "abc{1,3}" as match one, two, or three
occurrences of โabcโ, whereas ๐ณ๐น๐ฒ๐
interprets it as match โabโ
followed by one, two, or three occurrences of โcโ. The latter
is in agreement with the POSIX specification.
- The precedence of the โ^โ operator is different. ๐น๐ฒ๐
interโ
prets "^foo|bar" as match either โfooโ at the beginning of a
line, or โbarโ anywhere, whereas ๐ณ๐น๐ฒ๐
interprets it as match
either โfooโ or โbarโ if they come at the beginning of a line.
The latter is in agreement with the POSIX specification.
- The special table-size declarations such as โ%aโ supported by
๐น๐ฒ๐
are not required by ๐ณ๐น๐ฒ๐
scanners; ๐ณ๐น๐ฒ๐
ignores them.
- The name FLEX_SCANNER is #define'd so scanners may be written
for use with either ๐ณ๐น๐ฒ๐
or ๐น๐ฒ๐
. Scanners also include
YY_FLEX_MAJOR_VERSION and YY_FLEX_MINOR_VERSION indicating
which version of ๐ณ๐น๐ฒ๐
generated the scanner (for example, for
the 2.5 release, these defines would be 2 and 5, respectively).
The following ๐ณ๐น๐ฒ๐
features are not included in ๐น๐ฒ๐
or the POSIX
specification:
C++ scanners
%option
start condition scopes
start condition stacks
interactive/non-interactive scanners
yy_scan_string() and friends
yyterminate()
yy_set_interactive()
yy_set_bol()
YY_AT_BOL()
<<EOF>>
<*>
YY_DECL
YY_START
YY_USER_ACTION
YY_USER_INIT
#line directives
%{}'s around actions
multiple actions on a line
plus almost all of the ๐ณ๐น๐ฒ๐
flags. The last feature in the list
refers to the fact that with ๐ณ๐น๐ฒ๐
multiple actions can be placed on
the same line, separated with semi-colons, while with ๐น๐ฒ๐
, the folโ
lowing
foo handle_foo(); ++num_foos_seen;
is (rather surprisingly) truncated to
foo handle_foo();
๐ณ๐น๐ฒ๐
does not truncate the action. Actions that are not enclosed
in braces are simply terminated at the end of the line.
๐
๐๐๐๐
flex.skl Skeleton scanner. This file is only used when
building flex, not when ๐ณ๐น๐ฒ๐
executes.
lex.backup Backing-up information for the -๐ฏ flag (called
lฬฒeฬฒxฬฒ.ฬฒbฬฒcฬฒkฬฒ on some systems).
lex.yy.c Generated scanner (called lฬฒeฬฒxฬฒyฬฒyฬฒ.ฬฒcฬฒ on some sysโ
tems).
lex.yy.cc Generated C++ scanner class, when using -+.
<g++/FlexLexer.h> Header file defining the C++ scanner base class,
FฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ, and its derived class, yฬฒyฬฒFฬฒlฬฒeฬฒxฬฒLฬฒeฬฒxฬฒeฬฒrฬฒ.
/usr/lib/libl.* ๐ณ๐น๐ฒ๐
libraries. The /ฬฒuฬฒsฬฒrฬฒ/ฬฒlฬฒiฬฒbฬฒ/ฬฒlฬฒiฬฒbฬฒfฬฒlฬฒ.ฬฒ*ฬฒ libraries
are links to these. Scanners must be linked
using either -๐น๐น or -๐น๐ณ๐น.
๐๐๐๐ ๐๐๐๐๐๐
The ๐ณ๐น๐ฒ๐
utility exits 0 on success, and >0 if an error occurs.
๐๐๐๐๐๐๐๐๐๐๐
๐๐ฎ๐ฟ๐ป๐ถ๐ป๐ด, ๐ฟ๐๐น๐ฒ ๐ฐ๐ฎ๐ป๐ป๐ผ๐ ๐ฏ๐ฒ ๐บ๐ฎ๐๐ฐ๐ต๐ฒ๐ฑ Indicates that the given rule canโ
not be matched because it follows other rules that will always
match the same text as it. For example, in the following โfooโ
cannot be matched because it comes after an identifier "catch-all"
rule:
[a-z]+ got_identifier();
foo got_foo();
Using RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ in a scanner suppresses this warning.
๐๐ฎ๐ฟ๐ป๐ถ๐ป๐ด, -๐ ๐ผ๐ฝ๐๐ถ๐ผ๐ป ๐ด๐ถ๐๐ฒ๐ป ๐ฏ๐๐ ๐ฑ๐ฒ๐ณ๐ฎ๐๐น๐ ๐ฟ๐๐น๐ฒ ๐ฐ๐ฎ๐ป ๐ฏ๐ฒ ๐บ๐ฎ๐๐ฐ๐ต๐ฒ๐ฑ Means
that it is possible (perhaps only in a particular start condition)
that the default rule (match any single character) is the only one
that will match a particular input. Since -๐ was given, presumably
this is not intended.
๐ฟ๐ฒ๐ท๐ฒ๐ฐ๐_๐๐๐ฒ๐ฑ_๐ฏ๐๐_๐ป๐ผ๐_๐ฑ๐ฒ๐๐ฒ๐ฐ๐๐ฒ๐ฑ ๐๐ป๐ฑ๐ฒ๐ณ๐ถ๐ป๐ฒ๐ฑ
๐๐๐บ๐ผ๐ฟ๐ฒ_๐๐๐ฒ๐ฑ_๐ฏ๐๐_๐ป๐ผ๐_๐ฑ๐ฒ๐๐ฒ๐ฐ๐๐ฒ๐ฑ ๐๐ป๐ฑ๐ฒ๐ณ๐ถ๐ป๐ฒ๐ฑ These errors can occur at
compile time. They indicate that the scanner uses RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ or
๐๐๐บ๐ผ๐ฟ๐ฒ() but that ๐ณ๐น๐ฒ๐
failed to notice the fact, meaning that ๐ณ๐น๐ฒ๐
scanned the first two sections looking for occurrences of these
actions and failed to find any, but somehow they snuck in (via an
#include file, for example). Use โ%option rejectโ or โ%option
yymoreโ to indicate to ๐ณ๐น๐ฒ๐
that these features are really needed.
๐ณ๐น๐ฒ๐
๐๐ฐ๐ฎ๐ป๐ป๐ฒ๐ฟ ๐ท๐ฎ๐บ๐บ๐ฒ๐ฑ A scanner compiled with -๐ has encountered an
input string which wasn't matched by any of its rules. This error
can also occur due to internal problems.
๐๐ผ๐ธ๐ฒ๐ป ๐๐ผ๐ผ ๐น๐ฎ๐ฟ๐ด๐ฒ, ๐ฒ๐
๐ฐ๐ฒ๐ฒ๐ฑ๐ ๐๐๐๐๐๐ The scanner uses โ%arrayโ and one
of its rules matched a string longer than the YYLMAX constant (8K
bytes by default). The value can be increased by #define'ing
YYLMAX in the definitions section of ๐ณ๐น๐ฒ๐
input.
๐๐ฐ๐ฎ๐ป๐ป๐ฒ๐ฟ ๐ฟ๐ฒ๐พ๐๐ถ๐ฟ๐ฒ๐ -๐ด ๐ณ๐น๐ฎ๐ด ๐๐ผ ๐๐๐ฒ ๐๐ต๐ฒ ๐ฐ๐ต๐ฎ๐ฟ๐ฎ๐ฐ๐๐ฒ๐ฟ '๐
' The scanner
specification includes recognizing the 8-bit character โxโ and the
-๐ด flag was not specified, and defaulted to 7-bit because the -๐๐ณ
or -๐๐
table compression options were used. See the discussion of
the -๐ณ flag for details.
๐ณ๐น๐ฒ๐
๐๐ฐ๐ฎ๐ป๐ป๐ฒ๐ฟ ๐ฝ๐๐๐ต-๐ฏ๐ฎ๐ฐ๐ธ ๐ผ๐๐ฒ๐ฟ๐ณ๐น๐ผ๐ unput() was used to push back so
much text that the scanner's buffer could not hold both the pushed-
back text and the current token in yฬฒyฬฒtฬฒeฬฒxฬฒtฬฒ. Ideally the scanner
should dynamically resize the buffer in this case, but at present
it does not.
๐ถ๐ป๐ฝ๐๐ ๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ ๐ผ๐๐ฒ๐ฟ๐ณ๐น๐ผ๐, ๐ฐ๐ฎ๐ป'๐ ๐ฒ๐ป๐น๐ฎ๐ฟ๐ด๐ฒ ๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ ๐ฏ๐ฒ๐ฐ๐ฎ๐๐๐ฒ ๐๐ฐ๐ฎ๐ป๐ป๐ฒ๐ฟ ๐๐๐ฒ๐
๐๐๐๐๐๐ The scanner was working on matching an extremely large
token and needed to expand the input buffer. This doesn't work
with scanners that use RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ.
๐ณ๐ฎ๐๐ฎ๐น ๐ณ๐น๐ฒ๐
๐๐ฐ๐ฎ๐ป๐ป๐ฒ๐ฟ ๐ถ๐ป๐๐ฒ๐ฟ๐ป๐ฎ๐น ๐ฒ๐ฟ๐ฟ๐ผ๐ฟ--๐ฒ๐ป๐ฑ ๐ผ๐ณ ๐ฏ๐๐ณ๐ณ๐ฒ๐ฟ ๐บ๐ถ๐๐๐ฒ๐ฑ This can
occur in an scanner which is reentered after a long-jump has jumped
out (or over) the scanner's activation frame. Before reentering
the scanner, use:
yyrestart(yyin);
or, as noted above, switch to using the C++ scanner class.
๐๐ผ๐ผ ๐บ๐ฎ๐ป๐ ๐๐๐ฎ๐ฟ๐ ๐ฐ๐ผ๐ป๐ฑ๐ถ๐๐ถ๐ผ๐ป๐ ๐ถ๐ป <> ๐ฐ๐ผ๐ป๐๐๐ฟ๐๐ฐ๐! More start conditions
than exist were listed in a <> construct (so at least one of them
must have been listed twice).
๐๐๐ ๐๐๐๐
awk(1), sed(1), yacc(1)
John Levine, Tony Mason, and Doug Brown, Lฬฒeฬฒxฬฒ &ฬฒ Yฬฒaฬฒcฬฒcฬฒ, Oฬฒ'ฬฒRฬฒeฬฒiฬฒlฬฒlฬฒyฬฒ aฬฒnฬฒdฬฒ
Aฬฒsฬฒsฬฒoฬฒcฬฒiฬฒaฬฒtฬฒeฬฒsฬฒ, 2nd edition.
Alfred Aho, Ravi Sethi, and Jeffrey Ullman, Cฬฒoฬฒmฬฒpฬฒiฬฒlฬฒeฬฒrฬฒsฬฒ:ฬฒ Pฬฒrฬฒiฬฒnฬฒcฬฒiฬฒpฬฒlฬฒeฬฒsฬฒ,ฬฒ
Tฬฒeฬฒcฬฒhฬฒnฬฒiฬฒqฬฒuฬฒeฬฒsฬฒ aฬฒnฬฒdฬฒ Tฬฒoฬฒoฬฒlฬฒsฬฒ, Aฬฒdฬฒdฬฒiฬฒsฬฒoฬฒnฬฒ-ฬฒWฬฒeฬฒsฬฒlฬฒeฬฒyฬฒ, 1986, Describes the pattern-
matching techniques used by flex (deterministic finite automata).
๐๐๐๐๐๐๐๐๐
The ๐น๐ฒ๐
utility is compliant with the IEEE Std 1003.1-2008
(โPOSIX.1โ) specification, though its presence is optional.
The flags [-๐ณ๐ด๐๐ฏ๐๐ฑ๐
๐ณ๐ต๐๐ถ๐๐น๐ผ๐๐ฝ๐๐๐๐๐+?], [--๐ต๐ฒ๐น๐ฝ], and [--๐๐ฒ๐ฟ๐๐ถ๐ผ๐ป] are
extensions to that specification.
See also the IฬฒNฬฒCฬฒOฬฒMฬฒPฬฒAฬฒTฬฒIฬฒBฬฒIฬฒLฬฒIฬฒTฬฒIฬฒEฬฒSฬฒ WฬฒIฬฒTฬฒHฬฒ LฬฒEฬฒXฬฒ AฬฒNฬฒDฬฒ PฬฒOฬฒSฬฒIฬฒXฬฒ section, above.
๐๐๐๐๐๐๐
Vern Paxson, with the help of many ideas and much inspiration from
Van Jacobson. Original version by Jef Poskanzer. The fast table
representation is a partial implementation of a design done by Van
Jacobson. The implementation was done by Kevin Gong and Vern Paxโ
son.
Thanks to the many ๐ณ๐น๐ฒ๐
beta-testers, feedbackers, and contribuโ
tors, especially Francois Pinard, Casey Leedom, Robert Abramovitz,
Stan Adermann, Terry Allen, David Barker-Plummer, John Basrai, Neal
Becker, Nelson H.F. Beebe, bฬฒeฬฒnฬฒsฬฒoฬฒnฬฒ@ฬฒoฬฒdฬฒiฬฒ.ฬฒcฬฒoฬฒmฬฒ, Karl Berry, Peter A.
Bigot, Simon Blanchard, Keith Bostic, Frederic Brehm, Ian Brockโ
bank, Kin Cho, Nick Christopher, Brian Clapper, J.T. Conklin, Jason
Coughlin, Bill Cox, Nick Cropper, Dave Curtis, Scott David Daniels,
Chris G. Demetriou, Theo de Raadt, Mike Donahue, Chuck Doucette,
Tom Epperly, Leo Eskin, Chris Faylor, Chris Flatters, Jon Forrest,
Jeffrey Friedl, Joe Gayda, Kaveh R. Ghazi, Wolfgang Glunz, Eric
Goldman, Christopher M. Gould, Ulrich Grepel, Peer Griebel, Jan
Hajic, Charles Hemphill, NORO Hideo, Jarkko Hietaniemi, Scott Hofโ
mann, Jeff Honig, Dana Hudes, Eric Hughes, John Interrante, Ceriel
Jacobs, Michal Jaegermann, Sakari Jalovaara, Jeffrey R. Jones,
Henry Juengst, Klaus Kaempf, Jonathan I. Kamens, Terrence O Kane,
Amir Katz, kฬฒeฬฒnฬฒ@ฬฒkฬฒeฬฒnฬฒ.ฬฒhฬฒiฬฒlฬฒcฬฒoฬฒ.ฬฒcฬฒoฬฒmฬฒ, Kevin B. Kenny, Steve Kirsch, Winโ
fried Koenig, Marq Kole, Ronald Lamprecht, Greg Lee, Rohan Lenard,
Craig Leres, John Levine, Steve Liddle, David Loffredo, Mike Long,
Mohamed el Lozy, Brian Madsen, Malte, Joe Marshall, Bengt Martensโ
son, Chris Metcalf, Luke Mewburn, Jim Meyering, R. Alexander
Milowski, Erik Naggum, G.T. Nicol, Landon Noll, James Nordby, Marc
Nozell, Richard Ohnemus, Karsten Pahnke, Sven Panne, Roland Pesch,
Walter Pelissero, Gaumond Pierre, Esmond Pitt, Jef Poskanzer, Joe
Rahmeh, Jarmo Raiha, Frederic Raimbault, Pat Rankin, Rick Richardโ
son, Kevin Rodgers, Kai Uwe Rommel, Jim Roskind, Alberto Santini,
Andreas Scherer, Darrell Schiebel, Raf Schietekat, Doug Schmidt,
Philippe Schnoebelen, Andreas Schwab, Larry Schwimmer, Alex Siegel,
Eckehard Stolz, Jan-Erik Strvmquist, Mike Stump, Paul Stuart, Dave
Tallman, Ian Lance Taylor, Chris Thewalt, Richard M. Timoney, Jodi
Tsai, Paul Tuinenga, Gary Weik, Frank Whaley, Gerhard Wilhelms,
Kent Williams, Ken Yap, Ron Zellar, Nathan Zelle, David Zuhn, and
those whose names have slipped my marginal mail-archiving skills
but whose contributions are appreciated all the same.
Thanks to Keith Bostic, Jon Forrest, Noah Friedman, John Gilmore,
Craig Leres, John Levine, Bob Mulcahy, G.T. Nicol, Francois
Pinard, Rich Salz, and Richard Stallman for help with various disโ
tribution headaches.
Thanks to Esmond Pitt and Earle Horton for 8-bit character support;
to Benson Margulies and Fred Burke for C++ support; to Kent
Williams and Tom Epperly for C++ class support; to Ove Ewerlid for
support of NUL's; and to Eric Hughes for support of multiple bufโ
fers.
This work was primarily done when I was with the Real Time Systems
Group at the Lawrence Berkeley Laboratory in Berkeley, CA. Many
thanks to all there for the support I received.
Send comments to โจvฬฒeฬฒrฬฒnฬฒ@ฬฒeฬฒeฬฒ.ฬฒlฬฒbฬฒlฬฒ.ฬฒgฬฒoฬฒvฬฒโฉ.
๐๐๐๐
Some trailing context patterns cannot be properly matched and genโ
erate warning messages (dangerous trailing context). These are
patterns where the ending of the first part of the rule matches the
beginning of the second part, such as "zx*/xy*", where the โx*โ
matches the โxโ at the beginning of the trailing context. (Note
that the POSIX draft states that the text matched by such patterns
is undefined.)
For some trailing context rules, parts which are actually fixed-
length are not recognized as such, leading to the above mentioned
performance loss. In particular, parts using โ|โ or โ{n}โ (such as
"foo{3}") are always considered variable-length.
Combining trailing context with the special โ|โ action can result
in fixed trailing context being turned into the more expensive
variable trailing context. For example, in the following:
%%
abc |
xyz/def
Use of ๐๐ป๐ฝ๐๐() invalidates yytext and yyleng, unless the โ%arrayโ
directive or the -๐น option has been used.
Pattern-matching of NUL's is substantially slower than matching
other characters.
Dynamic resizing of the input buffer is slow, as it entails rescanโ
ning all the text matched so far by the current (generally huge)
token.
Due to both buffering of input and read-ahead, it is not possible
to intermix calls to <sฬฒtฬฒdฬฒiฬฒoฬฒ.ฬฒhฬฒ> routines, such as, for example,
๐ด๐ฒ๐๐ฐ๐ต๐ฎ๐ฟ(), with ๐ณ๐น๐ฒ๐
rules and expect it to work. Call ๐ถ๐ป๐ฝ๐๐()
instead.
The total table entries listed by the -๐ flag excludes the number
of table entries needed to determine what rule has been matched.
The number of entries is equal to the number of DFA states if the
scanner does not use RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ, and somewhat greater than the number
of states if it does.
RฬฒEฬฒJฬฒEฬฒCฬฒTฬฒ cannot be used with the -๐ณ or -๐
options.
The ๐ณ๐น๐ฒ๐
internal algorithms need documentation.
COSMOPOLITAN September 21, 2015 BSD
c91b3c5006