Understanding the order that the lexer tokenizes

[krum110487]

So, I seem to get some strange occurrences where sometimes the lexer will tokenize something lower on the list, instead of the one higher up.

How is the regex run, is it ran over the WHOLE document all at once?
Is the Order of the Regex from top to bottom?
Can two different tokens hit the same thing?
Can Regex determine ^ and $, but I am not aware if the ^ is the beginning of the line, or the file or what?
If something it found and tokenized, is it removed from being tokenized in the future or what?

if I have regex like (this is overly simplified)
[A-Z]+
[^\s]+

it seems like the second one sometimes grabs uppercase letters as well, I think this may be an issue with me not understanding the order and process the regex is called.

[alecthomas]

They're tried one after the other in order.

[krum110487]

Thank you for clarifying that piece, as for the other questions, is there a good way to debug what the lexer is doing, which items it is tagging? It seems like a black box sometimes and not acting as I expect it to when testing the regex in a regex tester manually.

[alecthomas]

Not to debug the internal state no, but I will typically use Next() against some small input sample and check the token types it's reporting against those returned by Symbols() to verify that it's doing what I think it's doing.