Memory usage increase

[arasic]

The RAM memory usage keeps increasing as events are ingested. I tried to deactivate caching, but nothing worked.

How to reproduce:

• I used the exact docker-compose sample from https://github.com/openobserve/openobserve
• To ingest logs, I used: https://openobserve.ai/docs/ingestion/logs/curl/ - more accurately the sample file with large number of events (3846 records).
• Monitored the RAM memory usage with ctop
• As I keeped inserting the events with curl command: curl http://localhost:5080/api/default/quickstart1/_json -i -u 'root@example.com:Complexpass#123' --data-binary "@k8slog_json.json" - the RAM memory was increasing with 100MB approx.
• I tried all env variables to turn off caching: https://openobserve.ai/docs/environment-variables/ to disable caching but it did not work.

For me, RAM usage is important, I do not care if READ on disk is more expensive.

[hengfeiyang]

Yes, we write data into memtable for better performance, and the default memtable limit is 256MB. so even the memory usage up to 512MB, it is also no problem. if you want to change this limit, just give it a small size, like:

ZO_MAX_FILE_SIZE_ON_DISK = 16
ZO_MAX_FILE_SIZE_IN_MEMORY = 64

But too small setting will let the query not efficient.