Generators

[p7g]

I recently refactored the interpreter such that calling cb_eval evaluates a function. This means that when a function returns, cb_eval also returns.

This design makes it easy (ish) to implement generators:

1. If a function contains a yield expression, calling it returns a generator object.
2. When this generator object is called, it is evaluated until it reaches a yield.
3. When a yield expression is encountered, cb_eval stores whatever state is needed to resume evaluation in the generator object, and then returns.
4. Any value provided to yield is returned from calling the generator object.
5. If an argument is passed when calling the generator object, it is put on the stack when the generator resumes.

Here is an example:

generator range(n) {
    for let i = 0; i < n; i = i + 1 {
        yield i
    }
}

The use of a generator keyword could help to implement this with a single-pass compiler, but might not be necessary. Javascript does not have a keyword, but similarly requires a function to be declared as a generator with function*. Python, however, does not require anything. A generator is simply any function that contains yield.

The bytecode for a yield expression could be as follows:

; evaluate the expression to be yielded, if there is no expression, the value is null
CONST_NULL
; at this instruction, cb_eval would return
YIELD
; if the result of yield is not used:
POP

In terms of grammar, yield could be easily parsed as a prefix unary operator.

Some challenges might be dealing with arguments to the generator. These will need to be stored in the generator object, in all likelihood. We don't want to need separate opcodes to handle retrieving arguments within a generator, so they will need to be pushed onto the stack every time the generator is resumed.

An approach could be to add a CB_VALUE_USERDATA type, which simple holds a void *, and then generators could be implemented in terms of builtin functions like this:

let gen = range(10);
for let i = next(gen); !done(gen); i = next(gen) {
    println(i);
} 

Since this requires a yield keyword, however, it might be worth adding it as a first-class feature of the language (i.e. CB_VALUE_GENERATOR type).