Smarter batching

[adamchalmers]

Right now, the batching algorithm is

• If an API request's output address is None, add it to a queue
• Run this queue in a batch (i.e. flush it):
  • Just before running an API request whose output address is Some is run
  • At the end of the program

But we can enhance this and batch a bit more.

We could still batch API requests whose output is written to an address (i.e. add them to the batch queue instead of executing them). We'd just have to add them to the current batch, and then flush it. That way the output address is still written to at the right time.

This would require modifying the batch queue. Currently it assumes all API responses get discarded. It'll need to instead track which API requests should be written where.