Request for feature: partial parsing

[pontaoski]

Description:

A new method that starts the parser as usual, but returns:

• (State, Resumption, Error)

State is either "Incomplete" (the structure isn't fully parsed) or "Complete" (the structure is fully parsed).
Resumption is something that will allow you to resume parsing with more tokens if State == Incomplete.

If it hits EOF while in the middle of parsing a structure, it pauses, returning something to allow you to resume the parsing from where it left off with new input instead of erroring.

Usecase:

REPLs.

Something like:

document := &ast{}
line := readline()
state, resumption, error := parser.ParsePartial(&document, line)

for {
  if err != nil {
    // input is malformed
  }
  if state == Complete {
    // evaluate code
  } else if state == Incomplete {
    line = readline()
    state, resumption, err = resumption.Resume(line)
  }
}

[alecthomas]

I do like the idea of incremental parsing. Most previous discussion has revolved around implementing a streaming lexer that blocks until the input is completely consumed. Would that satisfy your use case?

[pontaoski]

Ideally, I'd like the following things:

• being able to know if it's awaiting more input, so that I can signal such to the user with a new ... > in the repl
• being able to know where in the parser it's currently parsing (e.g. to make REPL print differently for an incomplete pattern match or a function)

If a streaming parser can do that, then sounds good to me. Though, it seems like I would need to utilise goroutines with that to get the desired result, when it should probably be a single-goroutine affair.

[ysmood]

It's also useful for IDE syntax highlighting.