clickhouse-backup server consumes memory when idle after first backup

[viktordebulat]

Why clickhouse-backup server container placed as a sidecar inside clickhouse server pod consumes memory when idle after first backup succeed. Using remote backups on gcs. When just started, container consumes tens of Mi. But after backup made and uploaded, it consumes couple Gi of memory whole day being idle.

Image: altinity/clickhouse-backup:2.4.30

Default config overrides:

  general:
    remote_storage: gcs
    backups_to_keep_local: -1
    backups_to_keep_remote: 7
    download_concurrency: 1
    upload_concurrency: 1
    restore_schema_on_cluster: "{cluster}"
  clickhouse:
    max_connections: 2
  gcs:
    path: backup/{replica}-{shard}
    client_pool_size: 3
  api:
    create_integration_tables: true

Other gcs and clickhouse creds related config set over env variables.

Any ideas how to deal with such issue? Whole pod restart after each backup not an option.

[Slach]

which clickhouse-backup version do you use?
please upgrade to 2.4.31
we have some unnecessary memory buffer allocations in some 2.4.x releases

[viktordebulat]

Bumped to 2.4.31 - the same picture. First it consumes almost nothing. Then, when upload started it consumes more (that's logical, ok) but after finished it consumes the same amount of memory, not increasing. The same picture on each node.

[viktordebulat]

@Slach Nope, it proceeded to grow. Meanwhile, on 1 node it grew up to 11 Gi (being idle), and on another node decreased from 2.68 to 1+ Gi

[Slach]

what mean WSS here?
how do you calculate this metric?

[viktordebulat]

WSS - Working Set Size, container metrics from cAdvisor
container_memory_working_set_bytes

[Slach]

most of allocations required during UPLOAD/DOWNLOAD inside remote storage related SDKs
i will appreciate with debuging memory allocations and help how to resolve it

[Slach]

what mean wss here the same as rss?
could you share curl http://127.0.0.1:7171/metrics | grep -i mem inside container which have grow memory related metrics?

[viktordebulat]

# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.500536e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 9.150332264e+09
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.548252e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 7.299674e+06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.865024e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.500536e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 7.2851456e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.070848e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 10180
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 7.2327168e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.7922304e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7084448142689362e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 7.309854e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 9600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 188800
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 310080
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.820376e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.49486e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.769472e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.769472e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 8.6925592e+07
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.5500032e+07
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.363279872e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19

[Slach]

process_resident_memory_bytes 3.5500032e+07

it means only 33.86 MiB

could you explain how exactly do you calculate memory on your graph?

[viktordebulat]

cAdvisor - container_memory_working_set_bytes
Now memory usage decreased to 4 Gi on 1 node (where 11 Gi was) and became twice bigger on another (from 1.3 to 2.7). No data transfer from clickhouse-backup container was during that time (backup uploaded long ago)

[Slach]

could you also run
curl -o heap.out http://127.0.0.1:7171//debug/pprof/heap?seconds=120

and share heap.out ?

[viktordebulat]

Failed to open the file heap.out: Permission denied

using inside podTemplate

  securityContext:
    fsGroup: 101
    runAsGroup: 101
    runAsUser: 101

Could it be an issue?

[Slach]

curl -o /tmp/heap.out http://127.0.0.1:7171//debug/pprof/heap?seconds=120

after it use kubectl cp to copy heap.out

[viktordebulat]

No file generated.

$ curl -o /tmp/heap.out http://127.0.0.1:7171//debug/pprof/heap?seconds=120
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

[Slach]

did you wait 120 seconds?

[viktordebulat]

It doesn't generate anything, request is ended right after its execution without response