Fixed max buffer limitation

[chernser]

Summary

Max buffer size can now be overridden by user

Checklist

Delete items not relevant to your PR:

• Unit and integration tests covering the common scenarios were added
• A human-readable description of the changes was provided to include in CHANGELOG
• For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Paultagoras]

This looks fine to me but I'll let Mark take a look as well - have you noticed a performance difference?

[chernser]

@Paultagoras
I see significant improvements when I set buffer almost equal to dataset size. With smaller buffers transfer takes about 4 secs.
With 750 mb buffer it is around 2 seconds. Whole dataset is generated in memory in 1600ms (1.6 s) - just for comparison.

I think when buffer is big is may fit whole data fast but if more space is needed than it will be costly to allocate another buffer. Smaller buffer have another side effect - they require more IO calls to push data to OS level.

So may be a good recommendation here is to use buffer size > estimated size of dataset to avoid expensive allocations.
Note for us: we need to research a buffer allocation algorithm that would be most effective in sending big datasets.

[Paultagoras]

@Paultagoras I see significant improvements when I set buffer almost equal to dataset size. With smaller buffers transfer takes about 4 secs. With 750 mb buffer it is around 2 seconds. Whole dataset is generated in memory in 1600ms (1.6 s) - just for comparison.

I think when buffer is big is may fit whole data fast but if more space is needed than it will be costly to allocate another buffer. Smaller buffer have another side effect - they require more IO calls to push data to OS level.

So may be a good recommendation here is to use buffer size > estimated size of dataset to avoid expensive allocations. Note for us: we need to research a buffer allocation algorithm that would be most effective in sending big datasets.

What sort of memory footprint did that have? I assume the logic of a smaller buffer is that it doesn't keep so much stored in RAM.

[chernser]

As I see it - if we have smaller buffers - we have to block more frequently to copy them to TCP layer. I'm currently talking about current implementation. But I have not tested it in this way.