Error message all of the sudden

[ishaiavrahami]

Description

Hello, last week i setup my db as a cluster (db1). everything worked. now upon checking my db logs im seeing this:

[error] 2023-11-30T07:45:35.402096Z couchdb@localhost <0.1495.0> -------- gen_server <0.1495.0> terminated with reason: killed
  last msg: redacted
     state: {state,#Ref<0.2775070371.1449263106.53394>,couch_replicator_doc_processor,nil,<<"_replicator">>,#Ref<0.2775070371.1449132034.53395>,nil,[],true}
    extra: []
[error] 2023-11-30T07:45:35.402174Z couchdb@localhost <0.1495.0> -------- gen_server <0.1495.0> terminated with reason: killed
  last msg: redacted
     state: {state,#Ref<0.2775070371.1449263106.53394>,couch_replicator_doc_processor,nil,<<"_replicator">>,#Ref<0.2775070371.1449132034.53395>,nil,[],true}
    extra: []
[error] 2023-11-30T07:45:35.402322Z couchdb@localhost <0.1495.0> -------- CRASH REPORT Process  (<0.1495.0>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_multidb_changes,init,['Argument__1']}, ancestors: [<0.428.0>,couch_replicator_sup,<0.393.0>], message_queue_len: 0, links: [], dictionary: [{io_priority,{system,<<"shards/80000000-ffffffff/_replicator.17003906...">>}}], trap_exit: true, status: running, heap_size: 1598, stack_size: 29, reductions: 35800
[error] 2023-11-30T07:45:35.402434Z couchdb@localhost <0.1495.0> -------- CRASH REPORT Process  (<0.1495.0>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_multidb_changes,init,['Argument__1']}, ancestors: [<0.428.0>,couch_replicator_sup,<0.393.0>], message_queue_len: 0, links: [], dictionary: [{io_priority,{system,<<"shards/80000000-ffffffff/_replicator.17003906...">>}}], trap_exit: true, status: running, heap_size: 1598, stack_size: 29, reductions: 35800
[error] 2023-11-30T07:45:35.406044Z couchdb@localhost <0.1186.0> -------- OS Process died with status: 143
[error] 2023-11-30T07:45:35.406269Z couchdb@localhost <0.1186.0> -------- gen_server <0.1186.0> terminated with reason: {exit_status,143}
  last msg: redacted
     state: {os_proc,"./bin/couchjs ./share/server/main.js",#Port<0.53>,fun couch_os_process:writejson/2,fun couch_os_process:readjson/1,5000,300000}
    extra: []
[error] 2023-11-30T07:45:35.406664Z couchdb@localhost <0.1186.0> -------- gen_server <0.1186.0> terminated with reason: {exit_status,143}
  last msg: redacted
     state: {os_proc,"./bin/couchjs ./share/server/main.js",#Port<0.53>,fun couch_os_process:writejson/2,fun couch_os_process:readjson/1,5000,300000}
    extra: []
[error] 2023-11-30T07:45:35.406816Z couchdb@localhost <0.1186.0> -------- CRASH REPORT Process  (<0.1186.0>) with 0 neighbors exited with reason: {exit_status,143} at gen_server:handle_common_reply/8(line:811) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_os_process,init,['Argument__1']}, ancestors: [<0.1142.0>], message_queue_len: 0, links: [<0.268.0>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 29, reductions: 60990
[error] 2023-11-30T07:45:35.409609Z couchdb@localhost <0.1186.0> -------- CRASH REPORT Process  (<0.1186.0>) with 0 neighbors exited with reason: {exit_status,143} at gen_server:handle_common_reply/8(line:811) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_os_process,init,['Argument__1']}, ancestors: [<0.1142.0>], message_queue_len: 0, links: [<0.268.0>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 29, reductions: 60990
[error] 2023-11-30T07:45:35.411364Z couchdb@localhost <0.2649.0> -------- OS Process died with status: 143
[error] 2023-11-30T07:45:35.411715Z couchdb@localhost <0.2649.0> -------- gen_server <0.2649.0> terminated with reason: {exit_status,143}
  last msg: redacted
     state: {os_proc,"./bin/couchjs ./share/server/main.js",#Port<0.107>,fun couch_os_process:writejson/2,fun couch_os_process:readjson/1,5000,300000}
    extra: []
[error] 2023-11-30T07:45:35.411928Z couchdb@localhost <0.2649.0> -------- gen_server <0.2649.0> terminated with reason: {exit_status,143}
  last msg: redacted
     state: {os_proc,"./bin/couchjs ./share/server/main.js",#Port<0.107>,fun couch_os_process:writejson/2,fun couch_os_process:readjson/1,5000,300000}
    extra: []
[error] 2023-11-30T07:45:35.412057Z couchdb@localhost <0.2649.0> -------- CRASH REPORT Process  (<0.2649.0>) with 0 neighbors exited with reason: {exit_status,143} at gen_server:handle_common_reply/8(line:811) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_os_process,init,['Argument__1']}, ancestors: [<0.2648.0>], message_queue_len: 0, links: [<0.268.0>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 29, reductions: 72903
[error] 2023-11-30T07:45:35.412169Z couchdb@localhost <0.2649.0> -------- CRASH REPORT Process  (<0.2649.0>) with 0 neighbors exited with reason: {exit_status,143} at gen_server:handle_common_reply/8(line:811) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_os_process,init,['Argument__1']}, ancestors: [<0.2648.0>], message_queue_len: 0, links: [<0.268.0>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 29, reductions: 72903
[error] 2023-11-30T07:45:35.454614Z couchdb@localhost <0.16834.16> -------- gen_server <0.16834.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16902.16>,couch_mrview_index,nil}
    extra: []
[error] 2023-11-30T07:45:35.454666Z couchdb@localhost <0.16834.16> -------- gen_server <0.16834.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16902.16>,couch_mrview_index,nil}
    extra: []
[error] 2023-11-30T07:45:35.454996Z couchdb@localhost <0.16834.16> -------- CRASH REPORT Process  (<0.16834.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_updater,init,['Argument__1']}, ancestors: [<0.16902.16>,<0.16899.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.455299Z couchdb@localhost <0.16834.16> -------- CRASH REPORT Process  (<0.16834.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_updater,init,['Argument__1']}, ancestors: [<0.16902.16>,<0.16899.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.455781Z couchdb@localhost <0.16930.16> -------- gen_server <0.16930.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16921.16>,couch_mrview_index,undefined}
    extra: []
[error] 2023-11-30T07:45:35.455820Z couchdb@localhost <0.16930.16> -------- gen_server <0.16930.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16921.16>,couch_mrview_index,undefined}
    extra: []
[error] 2023-11-30T07:45:35.456125Z couchdb@localhost <0.16930.16> -------- CRASH REPORT Process  (<0.16930.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_compactor,init,['Argument__1']}, ancestors: [<0.16921.16>,<0.16849.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.456492Z couchdb@localhost <0.16930.16> -------- CRASH REPORT Process  (<0.16930.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_compactor,init,['Argument__1']}, ancestors: [<0.16921.16>,<0.16849.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.456552Z couchdb@localhost <0.16856.16> -------- gen_server <0.16856.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16921.16>,couch_mrview_index,nil}
    extra: []
[error] 2023-11-30T07:45:35.456687Z couchdb@localhost <0.16856.16> -------- gen_server <0.16856.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16921.16>,couch_mrview_index,nil}
    extra: []
[error] 2023-11-30T07:45:35.456873Z couchdb@localhost <0.16856.16> -------- CRASH REPORT Process  (<0.16856.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_updater,init,['Argument__1']}, ancestors: [<0.16921.16>,<0.16849.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.457122Z couchdb@localhost <0.16856.16> -------- CRASH REPORT Process  (<0.16856.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_updater,init,['Argument__1']}, ancestors: [<0.16921.16>,<0.16849.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.457183Z couchdb@localhost <0.16833.16> -------- gen_server <0.16833.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16902.16>,couch_mrview_index,undefined}
    extra: []
[error] 2023-11-30T07:45:35.457324Z couchdb@localhost <0.16833.16> -------- gen_server <0.16833.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16902.16>,couch_mrview_index,undefined}
    extra: []
[error] 2023-11-30T07:45:35.462160Z couchdb@localhost <0.16833.16> -------- CRASH REPORT Process  (<0.16833.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_compactor,init,['Argument__1']}, ancestors: [<0.16902.16>,<0.16899.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.466219Z couchdb@localhost <0.16833.16> -------- CRASH REPORT Process  (<0.16833.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_compactor,init,['Argument__1']}, ancestors: [<0.16902.16>,<0.16899.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.466311Z couchdb@localhost <0.16853.16> -------- gen_server <0.16853.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16920.16>,couch_mrview_index,nil}
    extra: []
[error] 2023-11-30T07:45:35.469728Z couchdb@localhost <0.16853.16> -------- gen_server <0.16853.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16920.16>,couch_mrview_index,nil}
    extra: []
[error] 2023-11-30T07:45:35.469874Z couchdb@localhost <0.16853.16> -------- CRASH REPORT Process  (<0.16853.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_updater,init,['Argument__1']}, ancestors: [<0.16920.16>,<0.16919.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.470020Z couchdb@localhost <0.16853.16> -------- CRASH REPORT Process  (<0.16853.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_updater,init,['Argument__1']}, ancestors: [<0.16920.16>,<0.16919.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.470083Z couchdb@localhost <0.16923.16> -------- gen_server <0.16923.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16920.16>,couch_mrview_index,undefined}
    extra: []
[error] 2023-11-30T07:45:35.470126Z couchdb@localhost <0.16923.16> -------- gen_server <0.16923.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16920.16>,couch_mrview_index,undefined}
    extra: []
[error] 2023-11-30T07:45:35.470261Z couchdb@localhost <0.16923.16> -------- CRASH REPORT Process  (<0.16923.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_compactor,init,['Argument__1']}, ancestors: [<0.16920.16>,<0.16919.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.470358Z couchdb@localhost <0.16923.16> -------- CRASH REPORT Process  (<0.16923.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_compactor,init,['Argument__1']}, ancestors: [<0.16920.16>,<0.16919.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.470418Z couchdb@localhost <0.16831.16> -------- gen_server <0.16831.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16901.16>,couch_mrview_index,undefined}
    extra: []
[error] 2023-11-30T07:45:35.470458Z couchdb@localhost <0.16831.16> -------- gen_server <0.16831.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16901.16>,couch_mrview_index,undefined}
    extra: []
[error] 2023-11-30T07:45:35.470555Z couchdb@localhost <0.16831.16> -------- CRASH REPORT Process  (<0.16831.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_compactor,init,['Argument__1']}, ancestors: [<0.16901.16>,<0.16900.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.470638Z couchdb@localhost <0.16831.16> -------- CRASH REPORT Process  (<0.16831.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_compactor,init,['Argument__1']}, ancestors: [<0.16901.16>,<0.16900.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.470685Z couchdb@localhost <0.16832.16> -------- gen_server <0.16832.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16901.16>,couch_mrview_index,nil}
    extra: []
[error] 2023-11-30T07:45:35.470720Z couchdb@localhost <0.16832.16> -------- gen_server <0.16832.16> terminated with reason: killed
  last msg: redacted
     state: {st,<0.16901.16>,couch_mrview_index,nil}
    extra: []
[error] 2023-11-30T07:45:35.470834Z couchdb@localhost <0.16832.16> -------- CRASH REPORT Process  (<0.16832.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_updater,init,['Argument__1']}, ancestors: [<0.16901.16>,<0.16900.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427
[error] 2023-11-30T07:45:35.470922Z couchdb@localhost <0.16832.16> -------- CRASH REPORT Process  (<0.16832.16>) with 0 neighbors exited with reason: killed at gen_server:decode_msg/9(line:481) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_index_updater,init,['Argument__1']}, ancestors: [<0.16901.16>,<0.16900.16>], message_queue_len: 0, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 427

Steps to Reproduce

Expected Behaviour

Your Environment

• CouchDB version used: 3.3.2
• Browser name and version: VPS
• Operating system and version: Ubuntu 22.04

Additional Context

[nickva]

state: {os_proc,"./bin/couchjs ./share/server/main.js",#Port<0.53>,
[error] 2023-11-30T07:45:35.406044Z couchdb@localhost <0.1186.0> -------- OS Process died with status: 143
[error] 2023-11-30T07:45:35.406269Z couchdb@localhost <0.1186.0> -------- gen_server <0.1186.0> terminated with reason: {exit_status,143}

This indicates couch_js processes are being killed with the SIGTERM signal. 143 = 128 + 15 (SIGTERM). It's hard to say why that's happening or what's doing that. Maybe they are consuming too much memory or too many CPU resources.

You could try adjusting the values of [query_server_config] os_process_limit = 100 and [query_server_config] os_process_soft_limit = 100 to some other value, higher or lower and see if would make a difference. See the config reference: https://docs.couchdb.org/en/stable/config/query-servers.html#query_server_config for more details.

Alternatively, try adding more CPU or memory resources to your container/server.