Simpler and more efficient lexing algorithm.

First parsing tokens into a buffer, so no token needs to be scanned
twice.  Still... the scanpointer of StringScanner is only advanced
when #next method is called.