Grrrr.  Integrating backtracking and error recovery has proven much more difficult than I had expected.  :-(  The existing error recovery code works at the syntactic level, but in order to behave correctly in a backtracking context, it has to effect changes at the lexical level, as syntactic changes don’t survive a branch change.  Unfortunately, changing the underlying source in this way has proven non-trivial.

I’m currently working on moving lexing into the parser itself, so that it will be fully integrated with the backtracking system.  Doing so will make it a lot easier to make source changes for error recovery purposes, in that it will be possible to pre-calculate token examplars that can reliably produce the required type in the recovery branch, and that will still be present in other branches.  It also solves a design flaw that’s been bugging me for a while: in the separate-lexer design, token types could not be fully context-sensitive.  For instance, a keyword would always lex as a keyword if in a syntactic context where the keyword was valid — even if the following syntactic form required a different lexing.  Perhaps an example would be appropriate:

   if x == y then
      if = 10
      x = if
   end

In the second line, “if” is clearly meant as an identifier, not a keyword.  In the old code, it would always lex as a keyword in that context, because it could be the start of an if statement.  However, in the third line, the “if” would lex as an identifier, because the keyword “if” is not valid in that context.  Such arbitrary differences bother me.  The new design will allow backtracking to retry the lex with the second meaning, should the first meaning fail.

In any event, I had several false starts on this whole mess that wasted most of two weeks.  It only occurred to me that lexing could be done by the parser Friday night.  :-(  However, the requisite changes have been pushed out into three of the four layers of the compiler, with only the interpreter left to update.  Hopefully I’ll know some time Tuesday night if everything actually works — and performs well enough to be used.