SquirrelJME/SquirrelJME

# 2018/04/17

## 14:06

Okay so slight issue with the expanding tokenizer. I need to handle diverging
token sources. Because say if I perform a successful read of a split area I
need to set the source to use that divergent area. It also needs to support
peeking and such. So I think I can just go with another class and use a single
object for that. But the question is, what is the best way to do that? I
basically need a means to set success or failure. Will need to figure that
out. Also for a bunch of elements the major class structure I had does not
really make that much sense when it comes to inheriting, however it is much
better compared to before. I think what would be better is having elements of
the source file in an `element` package then having `lexical` just be the
parsers. That way I can structure things a bit better and just have parsers
be in other classes without much consideration. Actually maybe I do not have
to do this at all, just support marking and setting like input streams and
readers. I think that would be far better in the long run and likely easier
to debug. Also additionally this could just be done in another class that
provides tokens rather than modifying `ExpandingSource` even more. Basically
it will allow there to be sub-marks which can be closed as such. But there
is no way to tell if a close was a success or failure. That would still be
rather manual.

## 14:18

So just a change from `ExpandingSplit` to a source which can be marked and
such. They will all operate from the same object but it will just check the
close order and marking order as needed. But marks can just actually operate
on a single stream and there can just be multiple marks and sets. But each
sub-mark can just have a `commit()` method which just sets the parent stream
position to the given position. This would be taken on successful runs. But
that might not always be the case however. Just that this is a bit more
complicated than it seems.