Building Efficient Scanners

Issues:

• direct-encoding (lex is ten times slower)
• buffering (avoiding frequent/expensive tests)

Examples:

• Hand-coded: lcc
• Generated: re2c

Buffering

• Cannot hold entire file in memory.
• What if a token doesn't fit in the buffer?
  What if it crosses the boundary?
• Comments are a special case (tokens are unneeded).
• How does one test reaching the end of the buffer?

The lcc compiler

• small
• fast
• some optimization
• retargetable

• Scanner (not quite!) and parser are hand-written
• Code generator is generated

Buffering in lcc

Buffering:

• Use '\n' as sentinel;
• Move characters back and read when past 'fence';
• Always read 4096 characters.

(See lcc paper: page 3)

Keywords in lcc

• Direct-encoded decision tree for keywords
• Generated!

Re2C: A Scanner Generator

• Light-weight generation:

  
  #define YYCURSOR p
  unsigned char *scan_uint(unsigned char *p) {
  /*!re2c
      [0-9]+       {return p;}
      [\000-\377]  {return NULL;}
   */
  }
  

• As fast as hand-written C code.

Buffering in RE2C

• Left up to client!
• YYLIMIT checked at start and inside each loop.
• YYMARKER defined so characters can be moved.
• Paper gives one technique.

Even without sentinel, usually beats lcc.

Code Generation in RE2C

• Good heuristics for 'switch' code;
• Eliminating backtracking;
• ``Subroutine'' for comments.

Backtracking

• Even in a DFA!
  (due to longest match requirement)
• Characters are read multiple times.
• Affects buffering.