Memory Corruption in fuzzy_search_fields

[mgraczyk]

Description

I had a multi-search request failure while running typesense locally on my mac under docker.
The stack shows a memory corruption problem while trying to free something in the allocator.

E20240508 01:44:06.467671   114 backward.hpp:4200] Stack trace (most recent call last) in thread 114:
E20240508 01:44:06.467895   114 backward.hpp:4200] #17   Object "", at 0xffffffffffffffff, in
E20240508 01:44:06.467947   114 backward.hpp:4200] #16   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x2aaaab509a03, in __clone
E20240508 01:44:06.467993   114 backward.hpp:4200] #15   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x2aaaab478ac2, in
E20240508 01:44:06.468042   114 backward.hpp:4200] #14   Object "/opt/typesense-server", at 0x55555b0a6843, in execute_native_thread_routine
E20240508 01:44:06.468091   114 backward.hpp:4200] #13 | Source "include/threadpool.h", line 57, in operator()
E20240508 01:44:06.468132   114 backward.hpp:4200]       Source "/usr/include/c++/10/future", line 1592, in ThreadPool [0x555558586b1c]
E20240508 01:44:06.468165   114 backward.hpp:4200] #12 | Source "/usr/include/c++/10/future", line 1459, in _M_set_result
E20240508 01:44:06.468281   114 backward.hpp:4200]     | Source "/usr/include/c++/10/future", line 412, in call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, st
d::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*>
E20240508 01:44:06.468333   114 backward.hpp:4200]     | Source "/usr/include/c++/10/mutex", line 729, in __gthread_once
E20240508 01:44:06.468376   114 backward.hpp:4200]       Source "/usr/include/x86_64-linux-gnu/c++/10/bits/gthr-default.h", line 700, in _M_run [0x555558737a73]
E20240508 01:44:06.468421   114 backward.hpp:4200] #11   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x2aaaab47dee7, in
E20240508 01:44:06.468467   114 backward.hpp:4200] #10 | Source "/usr/include/c++/10/future", line 572, in operator()
E20240508 01:44:06.468510   114 backward.hpp:4200]       Source "/usr/include/c++/10/bits/std_function.h", line 622, in _M_do_set [0x555558585d32]
E20240508 01:44:06.468569   114 backward.hpp:4200] #9  | Source "/usr/include/c++/10/bits/std_function.h", line 292, in __invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, s
td::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_M_run<std::_Bind<HttpServer::proces
s_request(const std::shared_ptr<http_req>&, const std::shared_ptr<http_res>&, route_path*, const h2o_custom_req_handler_t*, bool)::<lambda()>()>, std::allocator<int>, void, {}>::<lambda()>, void>&>
E20240508 01:44:06.468605   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 115, in __invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std
::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_M_run<std::_Bind<HttpServer::process_
request(const std::shared_ptr<http_req>&, const std::shared_ptr<http_res>&, route_path*, const h2o_custom_req_handler_t*, bool)::<lambda()>()>, std::allocator<int>, void, {}>::<lambda()>, void>&>
E20240508 01:44:06.468631   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 60, in operator()
E20240508 01:44:06.468672   114 backward.hpp:4200]     | Source "/usr/include/c++/10/future", line 1397, in operator()
E20240508 01:44:06.468703   114 backward.hpp:4200]     | Source "/usr/include/c++/10/future", line 1456, in __invoke_r<void, std::_Bind<HttpServer::process_request(const std::shared_ptr<http_req>&, const std::shared_ptr<http_re
s>&, route_path*, const h2o_custom_req_handler_t*, bool)::<lambda()>()>&>
E20240508 01:44:06.468727   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 110, in __invoke_impl<void, std::_Bind<HttpServer::process_request(const std::shared_ptr<http_req>&, const std::shared_pt
r<http_res>&, route_path*, const h2o_custom_req_handler_t*, bool)::<lambda()>()>&>
E20240508 01:44:06.468760   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 60, in operator()<>
E20240508 01:44:06.468787   114 backward.hpp:4200]     | Source "/usr/include/c++/10/functional", line 499, in __call<void>
E20240508 01:44:06.468876   114 backward.hpp:4200]     | Source "/usr/include/c++/10/functional", line 416, in __invoke<HttpServer::process_request(const std::shared_ptr<http_req>&, const std::shared_ptr<http_res>&, route_path*
, const h2o_custom_req_handler_t*, bool)::<lambda()>&>
E20240508 01:44:06.469070   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 95, in __invoke_impl<void, HttpServer::process_request(const std::shared_ptr<http_req>&, const std::shared_ptr<http_res>&
, route_path*, const h2o_custom_req_handler_t*, bool)::<lambda()>&>
E20240508 01:44:06.469123   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 60, in operator()
E20240508 01:44:06.469156   114 backward.hpp:4200]       Source "src/http_server.cpp", line 706, in _M_invoke [0x555558739993]
E20240508 01:44:06.469192   114 backward.hpp:4200] #8    Source "src/core_api.cpp", line 737, in post_multi_search [0x5555586b3e7e]
E20240508 01:44:06.469235   114 backward.hpp:4200] #7    Source "src/collection_manager.cpp", line 1888, in do_search [0x555558658aa2]
E20240508 01:44:06.469255   114 backward.hpp:4200] #6    Source "src/collection.cpp", line 2352, in search [0x5555585dd851]
E20240508 01:44:06.469301   114 backward.hpp:4200] #5    Source "src/index.cpp", line 2257, in run_search [0x5555587a44cb]
E20240508 01:44:06.469332   114 backward.hpp:4200] #4    Source "src/index.cpp", line 3138, in search [0x55555879f7a0]
E20240508 01:44:06.469354   114 backward.hpp:4200] #3  | Source "src/index.cpp", line 3913, in ~sparse_hash_map
E20240508 01:44:06.469377   114 backward.hpp:4200]     | Source "include/sparsepp.h", line 4792, in ~sparse_hashtable
E20240508 01:44:06.469401   114 backward.hpp:4200]     | Source "include/sparsepp.h", line 3706, in ~sparsetable
E20240508 01:44:06.469429   114 backward.hpp:4200]       Source "include/sparsepp.h", line 3192, in fuzzy_search_fields [0x55555879750d]
E20240508 01:44:06.469455   114 backward.hpp:4200] #2  | Source "include/sparsepp.h", line 3141, in destruct
E20240508 01:44:06.469478   114 backward.hpp:4200]     | Source "include/sparsepp.h", line 2488, in _free_group
E20240508 01:44:06.469497   114 backward.hpp:4200]     | Source "include/sparsepp.h", line 2396, in ~pair
E20240508 01:44:06.469528   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/stl_pair.h", line 211, in ~vector
E20240508 01:44:06.469566   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/stl_vector.h", line 680, in _Destroy<std::__cxx11::basic_string<char>*, std::__cxx11::basic_string<char> >
E20240508 01:44:06.469594   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/alloc_traits.h", line 842, in _Destroy<std::__cxx11::basic_string<char>*>
E20240508 01:44:06.469619   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/stl_construct.h", line 185, in __destroy<std::__cxx11::basic_string<char>*>
E20240508 01:44:06.469645   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/stl_construct.h", line 152, in _Destroy<std::__cxx11::basic_string<char> >
E20240508 01:44:06.469671   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/stl_construct.h", line 140, in ~basic_string
E20240508 01:44:06.469695   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/basic_string.h", line 671, in _M_dispose
E20240508 01:44:06.469724   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/basic_string.h", line 240, in _M_destroy
E20240508 01:44:06.469741   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/basic_string.h", line 245, in deallocate
E20240508 01:44:06.469770   114 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/alloc_traits.h", line 492, in deallocate
E20240508 01:44:06.469795   114 backward.hpp:4200]       Source "/usr/include/c++/10/ext/new_allocator.h", line 139, in _free_groups [0x5555585f9b43]
E20240508 01:44:06.469822   114 backward.hpp:4200] #1  | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/src/jemalloc_cpp.cpp", line 200, in
sizedDeleteImpl
E20240508 01:44:06.469841   114 backward.hpp:4200]       Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/src/jemalloc_cpp.cpp", line 195, in
operator delete [0x55555a2f8d41]
E20240508 01:44:06.469867   114 backward.hpp:4200] #0  | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/src/jemalloc.c", line 3924, in isfre
e
E20240508 01:44:06.469895   114 backward.hpp:4200]     | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/src/jemalloc.c", line 2982, in isdal
loct
E20240508 01:44:06.469969   114 backward.hpp:4200]     | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/include/jemalloc/internal/jemalloc_i
nternal_inlines_c.h", line 133, in arena_sdalloc
E20240508 01:44:06.470000   114 backward.hpp:4200]     | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/include/jemalloc/internal/arena_inli
nes_b.h", line 421, in arena_dalloc_large
E20240508 01:44:06.470023   114 backward.hpp:4200]     | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/include/jemalloc/internal/arena_inli
nes_b.h", line 291, in emap_edata_lookup
E20240508 01:44:06.470057   114 backward.hpp:4200]     | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/include/jemalloc/internal/emap.h", l
ine 229, in rtree_read
E20240508 01:44:06.470088   114 backward.hpp:4200]     | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/include/jemalloc/internal/rtree.h",
line 437, in rtree_leaf_elm_read
E20240508 01:44:06.470113   114 backward.hpp:4200]     | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/include/jemalloc/internal/rtree.h",
line 242, in rtree_leaf_elm_bits_read
E20240508 01:44:06.470144   114 backward.hpp:4200]     | Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/include/jemalloc/internal/rtree.h",
line 182, in atomic_load_p
E20240508 01:44:06.470225   114 backward.hpp:4200]       Source "/root/.cache/bazel/_bazel_root/45fa99e3ef9c0d8b8591cceacfe70dd5/sandbox/linux-sandbox/232/execroot/__main__/external/jemalloc/include/jemalloc/internal/atomic.h",
 line 83, in je_sdallocx_default [0x55555a2788b2]
Segmentation fault (Address not mapped to object [0x1433b0])

Steps to reproduce

Unfortunately I can't repro

Expected Behavior

No crash, no segfault

Actual Behavior

Segfault while trying to free the sparse_hashtable in fuzzy_search_fields.

Metadata

Typesense Version:
v26.0

OS:
macOS with ARM M2, but typesense is running under docker with target linux/amd64