unchanging dat file when applying to official ClickHouse binaries #83
Comments
|
Looks like they're using their own So it's expected that it doesn't work. This kind of binaries are currently only supported for Rust programs using jemalloc. If you'd recompile CH and disable this then profiling should work. |
|
much thanks for the prompt response! I will try rebuilding a CH with linkable malloc. so far their we'll try their guide next, and also try rebuilding without both TCMALLOC & their jemalloc |
Does this build use |
|
i guess so. here's a build with here's how we build it in CH's builder image (i'll link their page here later):
cd docker/packager/
# change defines here
export CMAKE_FLAGS="-DENABLE_JEMALLOC=0 -DENABLE_TCMALLOC=0"
#
./packager --cache ccache --output-dir /root/ch-build --package-type binary --compiler clang-13 --docker-image-version 39450-amd64output dir will contain the big |
|
Okay, that's interesting. It is crashing, which definitely shouldn't happen. This line might be related: It might be because it's all done under docker (I do somewhat support running under docker, but probably not every corner case), or it's because CH might have used a weird linker and/or linker args when linking itself. You could maybe try to run it with |
we only build the binary in docker (base is ubuntu 20.04), to mimick CH's packaging method. we run the resulting file on-host (debian11). i too like to avoid any complication from docker infrastructure, if possible. here i have a backtrace (under gdb shows the same, shadow-stack=on/off is the same). it's something to do with avx2 memset when defining(?) error codes. http://ix.io/47gW it doesn't look bytehound-related somehow.
I tried recording with rr too. under it, mmap
would this be similar in shape to thanks in advance! |
Yes, call the real function and fixup the size (the allocations are bigger by default since they also contain a tag which Bytehound uses to track the allocations, but we can't let the original app know about this extra memory as it then might try to overwrite it)
Assuming the backtrace is correct, it's possible it's related. If you look at the lower frames you can see that this crash happens before
|
|
I'm also trying to test bytehound with ClickHouse and found exactly the same issues presented here. |
|
sorry, we have no further progress. the CH people fixed our issue with jemalloc diff. I tried adding |
(apologies for sparse details, I'll copy them more as i can formulate them concretely)
we're trying out bytehound to analyze a memleak. However, the dat file always come out at 190MB, and have no update after CH finished starting up. Furthermore, the file has
0sruntime,0Ballocation.We run on debian11, but CH builds are mostly self-contained. There's a few mode of starting up CH, you can try a one-shot command this way:
Later CH versions ignore
LD_PRELOADbtw, so we'll have to use 22.3 here.Flushingdebug log:
final sql & bytehound output:
dat file is not updated beyond 2 seconds
stat -c '%Y %s' memory-profiling_clickhouse_1660033877_1202.dat 1660033879 198359119and it contains no info
we would love to use bytehound's USR1 signaling to catch the relevant, long-term memleak. do you have any suggestion to gather more info?
we also tried
LD_PRELOADboth bytehound.so & system libjemalloc.so.2 at the same time, but it segfaults/aborts (details TBD). further more, in interactive mode (egclickhouse localwithout-q), it aborts right away too.thanks for the tooling!
The text was updated successfully, but these errors were encountered: