not found api: malloc

[tst-basilhuffman]

I am trying to profile a init.d daemon and am having issues. Of course, trying to run: memleax <pid> will not work. When I run as superuser i.e. sudo memleax <pid>, I get the following message:

not found api: malloc

Is this recoverable at all?

[WuBingzheng]

It seem that memleax does not find malloc symbol.

Would you like run the following commands on your system and paste the output here? Assume a.out is your target program.

$ ldd a.out
    linux-vdso.so.1 =>  (0x00007ffd079e9000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efffcbf7000)
    /lib64/ld-linux-x86-64.so.2 (0x00007efffcfc1000)
$ readelf -s /lib/x86_64-linux-gnu/libc.so.6 | grep -w malloc
  1185: 0000000000084130   414 FUNC    GLOBAL DEFAULT   13 malloc@@GLIBC_2.2.5

[tst-basilhuffman]

It is a daemon process.

$ ldd /opt/google/bin/gesystemmanager linux-vdso.so.1 => (0x00007fff98d4f000) libgegdal.so => /opt/google/bin/../lib/libgegdal.so (0x00007f950735d000) libgeraster.so => /opt/google/bin/../lib/libgeraster.so (0x00007f950712a000) libgesearchtabs.so => /opt/google/bin/../lib/libgesearchtabs.so (0x00007f9506f1b000) libgenet.so => /opt/google/bin/../lib/libgenet.so (0x00007f9506d13000) libgepublish.so => /opt/google/bin/../lib/libgepublish.so (0x00007f9506aeb000) libgefconfigutil.so => /opt/google/bin/../lib/libgefconfigutil.so (0x00007f95068ce000) libgeindex_r.so => /opt/google/bin/../lib/libgeindex_r.so (0x00007f95066aa000) libgeutil.so => /opt/google/bin/../lib/libgeutil.so (0x00007f9506470000) libgecommon.so => /opt/google/bin/../lib/libgecommon.so (0x00007f9506205000) libgexml.so => /opt/google/bin/../lib/libgexml.so (0x00007f9505ffe000) libgemiscconfig.so => /opt/google/bin/../lib/libgemiscconfig.so (0x00007f9505dee000) libqt-mt.so.3 => /opt/google/bin/../qt/lib/libqt-mt.so.3 (0x00007f95053ee000) libxerces-c-3.1.so => /lib64/libxerces-c-3.1.so (0x00007f9504e2c000) libgeos-3.4.2.so => /lib64/libgeos-3.4.2.so (0x00007f9504a8b000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f9504784000) libm.so.6 => /lib64/libm.so.6 (0x00007f9504482000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f950426b000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f950404f000) libc.so.6 => /lib64/libc.so.6 (0x00007f9503c82000) libgdal.so => /opt/google/bin/../lib/../lib/libgdal.so (0x00007f9502c24000) libgeautoingest.so => /opt/google/bin/../lib/../lib/libgeautoingest.so (0x00007f9502732000) libcurl.so.4 => /opt/google/bin/../lib/../lib/libcurl.so.4 (0x00007f95024c5000) libdbmanifest.so => /opt/google/bin/../lib/../lib/libdbmanifest.so (0x00007f9502297000) libgeconfigutil.so => /opt/google/bin/../lib/../lib/libgeconfigutil.so (0x00007f950208e000) libgeprotobuf.so => /opt/google/bin/../lib/../lib/libgeprotobuf.so (0x00007f9501dbd000) libz.so.1 => /lib64/libz.so.1 (0x00007f9501ba6000) libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007f9501951000) libpng12.so.0 => /lib64/libpng12.so.0 (0x00007f950172a000) libmng.so.1 => /lib64/libmng.so.1 (0x00007f95014b7000) libGL.so.1 => /lib64/libGL.so.1 (0x00007f9501245000) libXmu.so.6 => /lib64/libXmu.so.6 (0x00007f9501029000) libXrender.so.1 => /lib64/libXrender.so.1 (0x00007f9500e1e000) libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007f9500c13000) libXcursor.so.1 => /lib64/libXcursor.so.1 (0x00007f9500a07000) libXinerama.so.1 => /lib64/libXinerama.so.1 (0x00007f9500804000) libXft.so.2 => /lib64/libXft.so.2 (0x00007f95005ee000) libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f9500347000) libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007f950010a000) libXext.so.6 => /lib64/libXext.so.6 (0x00007f94ffef8000) libX11.so.6 => /lib64/libX11.so.6 (0x00007f94ffbb9000) libSM.so.6 => /lib64/libSM.so.6 (0x00007f94ff9b1000) libICE.so.6 => /lib64/libICE.so.6 (0x00007f94ff795000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f94ff590000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f94ff376000) /lib64/ld-linux-x86-64.so.2 (0x000055a6e984f000) libgeos_c.so.1 => /lib64/libgeos_c.so.1 (0x00007f94ff14e000) libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f94fef24000) libopenjp2.so.7 => /opt/google/bin/../lib/../lib/../lib/libopenjp2.so.7 (0x00007f94fecca000) libogdi.so.3 => /lib64/libogdi.so.3 (0x00007f94feaa8000) librt.so.1 => /lib64/librt.so.1 (0x00007f94fe8a0000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f94fe63e000) libltidsdk.so => /opt/MrSID_DSDK-9.5.4.4703-rhel6.x86-64.gcc482/Raster_DSDK/lib/libltidsdk.so (0x00007f94fde92000) libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f94fdb28000) libssl.so.1.0.0 => /opt/google/bin/../lib/../lib/../lib/libssl.so.1.0.0 (0x00007f94fd8b6000) libcrypto.so.1.0.0 => /opt/google/bin/../lib/../lib/../lib/libcrypto.so.1.0.0 (0x00007f94fd469000) liblber-2.4.so.2 => /opt/google/bin/../lib/../lib/../lib/liblber-2.4.so.2 (0x00007f94fd25c000) libldap-2.4.so.2 => /opt/google/bin/../lib/../lib/../lib/libldap-2.4.so.2 (0x00007f94fd01e000) libserverdb.so => /opt/google/bin/../lib/../lib/../lib/libserverdb.so (0x00007f94fce19000) libgedbroot.so => /opt/google/bin/../lib/../lib/../lib/libgedbroot.so (0x00007f94fcb50000) libcap.so.2 => /lib64/libcap.so.2 (0x00007f94fc94b000) libgtest.so.0 => /lib64/libgtest.so.0 (0x00007f94fc706000) libxcb-dri3.so.0 => /lib64/libxcb-dri3.so.0 (0x00007f94fc503000) libxcb-xfixes.so.0 => /lib64/libxcb-xfixes.so.0 (0x00007f94fc2fb000) libxcb-present.so.0 => /lib64/libxcb-present.so.0 (0x00007f94fc0f7000) libxcb-sync.so.1 => /lib64/libxcb-sync.so.1 (0x00007f94fbef0000) libxshmfence.so.1 => /lib64/libxshmfence.so.1 (0x00007f94fbced000) libglapi.so.0 => /lib64/libglapi.so.0 (0x00007f94fbabc000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f94fb895000) libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007f94fb692000) libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007f94fb48b000) libX11-xcb.so.1 => /lib64/libX11-xcb.so.1 (0x00007f94fb289000) libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f94fb061000) libxcb-glx.so.0 => /lib64/libxcb-glx.so.0 (0x00007f94fae45000) libxcb-dri2.so.0 => /lib64/libxcb-dri2.so.0 (0x00007f94fac40000) libXxf86vm.so.1 => /lib64/libXxf86vm.so.1 (0x00007f94faa3a000) libdrm.so.2 => /lib64/libdrm.so.2 (0x00007f94fa828000) libXt.so.6 => /lib64/libXt.so.6 (0x00007f94fa5c1000) libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f94fa3bb000) libproj.so.0 => /lib64/libproj.so.0 (0x00007f94fa169000) libtbb.so.2 => /opt/MrSID_DSDK-9.5.4.4703-rhel6.x86-64.gcc482/Raster_DSDK/lib/libtbb.so.2 (0x00007f94f9f34000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f94f9d0e000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f94f9af4000) libattr.so.1 => /lib64/libattr.so.1 (0x00007f94f98ef000) libXau.so.6 => /lib64/libXau.so.6 (0x00007f94f96ea000)

[tst-basilhuffman]

Sorry it is so long.

[WuBingzheng]

How about the second command?

readelf -s /lib64/libc.so.6 | grep -w malloc

[tst-basilhuffman]

`$ readelf -s /lib64/libc.so.6 | grep malloc

444: 0000000000085800 214 FUNC GLOBAL DEFAULT 12 __libc_malloc@@GLIBC_2.2.5

551: 0000000000087dd0 271 FUNC GLOBAL DEFAULT 12 malloc_info@@GLIBC_2.10

818: 0000000000087b90 450 FUNC WEAK DEFAULT 12 malloc_stats@@GLIBC_2.2.5

990: 0000000000085a20 507 FUNC WEAK DEFAULT 12 malloc_get_state@@GLIBC_2.2.5

1087: 00000000003c6740 8 OBJECT WEAK DEFAULT 33 __malloc_hook@@GLIBC_2.2.5

1180: 0000000000085800 214 FUNC GLOBAL DEFAULT 12 malloc@@GLIBC_2.2.5

1214: 00000000000864c0 252 FUNC WEAK DEFAULT 12 malloc_usable_size@@GLIBC_2.2.5

1465: 0000000000087810 624 FUNC WEAK DEFAULT 12 malloc_trim@@GLIBC_2.2.5

1786: 00000000003c8a40 8 OBJECT WEAK DEFAULT 34 __malloc_initialize_hook@@GLIBC_2.2.5

2084: 0000000000086fe0 1286 FUNC WEAK DEFAULT 12 malloc_set_state@@GLIBC_2.2.5

130: 00000000003c7ab0 4 OBJECT LOCAL DEFAULT 34 cache_malloced

1052: 0000000000000000 0 FILE LOCAL DEFAULT ABS malloc.c

1068: 000000000007f520 248 FUNC LOCAL DEFAULT 12 malloc_printerr

1101: 0000000000080500 2396 FUNC LOCAL DEFAULT 12 malloc_consolidate

1120: 0000000000081910 2010 FUNC LOCAL DEFAULT 12 sysmalloc

1123: 00000000000820f0 3852 FUNC LOCAL DEFAULT 12 _int_malloc

1132: 0000000000083000 282 FUNC LOCAL DEFAULT 12 malloc_check

1174: 00000000003c8ad0 8 OBJECT LOCAL DEFAULT 34 save_malloc_hook

1176: 00000000000858e0 305 FUNC LOCAL DEFAULT 12 malloc_atfork

1181: 00000000003c8a70 4 OBJECT LOCAL DEFAULT 34 disallow_malloc_check

1182: 00000000003c8a80 4 OBJECT LOCAL DEFAULT 34 using_malloc_checking

1197: 0000000000086790 935 FUNC LOCAL DEFAULT 12 ptmalloc_init.part.7

1198: 0000000000086b40 21 FUNC LOCAL DEFAULT 12 ptmalloc_init

1199: 0000000000086b60 60 FUNC LOCAL DEFAULT 12 malloc_hook_ini

1220: 0000000000088c20 495 FUNC LOCAL DEFAULT 12 mallochook

1222: 00000000003c8ba0 8 OBJECT LOCAL DEFAULT 34 old_malloc_hook

1241: 00000000003c8bf0 8 OBJECT LOCAL DEFAULT 34 tr_old_malloc_hook

1242: 0000000000089c60 189 FUNC LOCAL DEFAULT 12 tr_mallochook

1254: 00000000003c8c10 8 OBJECT LOCAL DEFAULT 34 malloc_trace_buffer

3478: 0000000000085610 174 FUNC LOCAL DEFAULT 12 __malloc_fork_unlock_pare

3813: 00000000003c6170 4 OBJECT LOCAL DEFAULT 33 __libc_malloc_initialized

3876: 0000000000086fe0 1286 FUNC LOCAL DEFAULT 12 __malloc_set_state

4197: 00000000000864c0 252 FUNC LOCAL DEFAULT 12 __malloc_usable_size

4255: 0000000000087810 624 FUNC LOCAL DEFAULT 12 __malloc_trim

4498: 0000000000085800 214 FUNC LOCAL DEFAULT 12 __GI___libc_malloc

4697: 0000000000085a20 507 FUNC LOCAL DEFAULT 12 __malloc_get_state

4735: 0000000000087b90 450 FUNC LOCAL DEFAULT 12 __malloc_stats

4851: 00000000000856c0 193 FUNC LOCAL DEFAULT 12 __malloc_fork_unlock_chil

5025: 0000000000085790 107 FUNC LOCAL DEFAULT 12 __malloc_check_init

5053: 0000000000085500 267 FUNC LOCAL DEFAULT 12 __malloc_fork_lock_parent

5547: 0000000000085800 214 FUNC LOCAL DEFAULT 12 __malloc

5685: 00000000003c8a40 8 OBJECT WEAK DEFAULT 34 __malloc_initialize_hook

5767: 0000000000086fe0 1286 FUNC WEAK DEFAULT 12 malloc_set_state

6067: 0000000000085800 214 FUNC GLOBAL DEFAULT 12 malloc

6077: 0000000000087dd0 271 FUNC GLOBAL DEFAULT 12 malloc_info

6190: 0000000000087810 624 FUNC WEAK DEFAULT 12 malloc_trim

6735: 00000000003c6740 8 OBJECT WEAK DEFAULT 33 __malloc_hook

6827: 00000000000864c0 252 FUNC WEAK DEFAULT 12 malloc_usable_size

7306: 0000000000085a20 507 FUNC WEAK DEFAULT 12 malloc_get_state

7323: 0000000000085800 214 FUNC GLOBAL DEFAULT 12 __libc_malloc

7492: 0000000000087b90 450 FUNC WEAK DEFAULT 12 malloc_stats`

[WuBingzheng]

There is malloc. I do not know why memleax does not find it.
Coud you add some debug print in source code and try it again?

diff --git a/symtab.c b/symtab.c
index f1b3df0..c20cc85 100644
--- a/symtab.c
+++ b/symtab.c
@@ -62,6 +62,7 @@ static int symtab_build_section(Elf *elf, Elf_Scn *section,
                struct symbol_s *sym = array_push(&g_symbol_table);

                sym->name = strdup(elf_strptr(elf, shdr.sh_link, (size_t)esym.st_name));
+               printf("symbol: %s\n", sym->name);

                sym->address = esym.st_value - base_addr + offset;
                sym->size = esym.st_size;
@@ -93,6 +94,7 @@ static uintptr_t symtab_elf_base(Elf *elf)

 static int symtab_build_file(const char *path, uintptr_t start, uintptr_t end)
 {
+       printf("symtab_build_file: %s\n", path);
        /* open file */
        int fd = open(path, O_RDONLY);
        if (fd < 0) {

[tst-basilhuffman]

I will check probably tomorrow morning (eastern us time) but could this possibly be related to differing PATH between user and super user?

[WuBingzheng]

I do not think it's related to users, but I am not sure about it.

[tst-basilhuffman]

It was a fairly large file, so check my repo for outfile

[WuBingzheng]

It seems that memleax does not load symbols from libc.so.
Could you patch again and show me the result?

diff --git a/symtab.c b/symtab.c
index f1b3df0..1397b7d 100644
--- a/symtab.c
+++ b/symtab.c
@@ -93,6 +93,7 @@ static uintptr_t symtab_elf_base(Elf *elf)

 static int symtab_build_file(const char *path, uintptr_t start, uintptr_t end)
 {
+       printf("symtab_build_file: %s\n", path);
        /* open file */
        int fd = open(path, O_RDONLY);
        if (fd < 0) {
@@ -140,6 +141,7 @@ void symtab_build(pid_t pid)
        size_t start, end;
        int exe_self;
        while ((path = proc_maps(pid, &start, &end, &exe_self)) != NULL) {
+       printf("path: %s\n", path);
                debug_try_init(path, exe_self);
                while ((debugp = debug_try_get()) != NULL) {
                        if (symtab_build_file(debugp, start, end) > 0) {

[WuBingzheng]

BTW, there is another thing: memleax now has a bug that it can not work with multi-thread target program.
Is your program multi-thread?

This is no related with this issue.

[tst-basilhuffman]

There is no difference between the issue. Yes, the target is multi-threaded

[WuBingzheng]

So memleax can not be used for your program.
Maybe you could try valgrind or some compile-time override libraries.