The New Parser Tools
The New Parser Tools is a complete rewrite of the old parser tools (also avalable on github). Core utilities like TokenScanner
and TokenList
are simplified and made easier to use.
The TokenScanner
does not implement TokenList
like in the old parser tools, but instead returns an instance of one, thus removing need to create a new scanner every time something is parsed.
A minimalistic scanner specifies all operator characters and other separators, which are then used to separate tokens.
TokenScanner scanner = new TokenScanner()
.addOperators("+-*/%()<>")
.addOperatorRule("==")
.addOperatorRule("!=")
.addOperatorRule("<=")
.addOperatorRule(">=")
.appendOnEOF("<EOF>");
By default TokenScanner
separates all punctuation characters from all other characters. If this is not needed, it can be turned of with .separateIdentifiersAndPunctuation(false)
.
A token list object is redesigned to provide more methods, thus moving the responsibility of some token related tasks from the parser class to the token list.
The core methods of TokenList
are still .next()
and .seek()
. New methods include .isNext()
, .nextString()
, .seekString()
and .accept()
.
public Operation parseInstruction(TokenList tl) {
if (tl.isNext("print")) {
tl.accept("print");
String var = tl.nextString();
return () -> System.out.print(variables.get(var));
}
else if (tl.isNext("read")) {
tl.accept("read");
String var = tl.nextString();
return () -> variables.put(var, System.in.read());
}
else {
String command = tl.next();
String var = tl.next();
return () -> commands.get(command).consume(var);
}
}
To make it easier to implement a short parser for a programming language, Nept includes an operator precedence parsing class. For complete example, see OPPExample.java
.
/* Declares the operator library – all operators use parsePrimary() as their RHS parser */
OperatorLibrary<Integer> library = new OperatorLibrary<>(tl -> parsePrimary(tl));
/* Declares the operators*/
library.add("+", (a, b) -> a + b);
library.add("-", (a, b) -> a - b);
library.increaseLevel();
library.add("*", (a, b) -> a * b);
library.add("/", (a, b) -> a / b);
/* Declares the OPP*/
OperatorPrecedenceParser<Integer> opparser = new OperatorPrecedenceParser<>(library);