Streaming API for RPC

[tejorupan]

Gorilla RPC does not have any provision to support streaming.

Use-case (1) is retrieving 1000s of records of data from a database using SQL, in a loop. We may not know how many rows are going to be retrieved nor we would like to store all the records in memory and do the marshalling of the whole data at once, which may not be memory efficient.

Use case (2) is to transmit bulk data, for example the contents of a file, whose size is several mega/giga bytes.

For the above mentioned cases, we avoid the RPC layer and directly work on the underlying network connection.

One interesting solution for streaming rpc is https://github.com/youtube/vitess/tree/master/go/rpcplus‎

We need a provision to process data in chunks (streaming mode) instead of waiting for the whole datastructure to be populated first.