crash: free(): invalid pointer #414

hazcod · 2021-05-17T14:44:51Z

Very odd, but I have a Docker container on Apple Silicon (M1) which crashes when I import google.cloud.
It works when I remove the import line.

Environment details

OS type and version: python:3.9.5-slim-buster on Docker
google-cloud-pubsub version: 2.8.6

Steps to reproduce

from google.cloud import pubsub_v1

Code example

Stack trace

free(): invalid pointer

strace excerpt

openat(AT_FDCWD, "/lib/aarch64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 6
read(6, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\220\36\0\0\0\0\0\0"..., 832) = 832
fstat(6, {st_mode=S_IFREG|0644, st_size=31576, ...}) = 0
mmap(NULL, 95016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 6, 0) = 0xffffb039f000
mprotect(0xffffb03a6000, 61440, PROT_NONE) = 0
mmap(0xffffb03b5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 6, 0x6000) = 0xffffb03b5000
close(6)                                = 0
mprotect(0xffffb03b5000, 4096, PROT_READ) = 0
mprotect(0xffffb0acf000, 167936, PROT_READ) = 0
futex(0xffffb0b32070, FUTEX_WAKE_PRIVATE, 2147483647) = 0
writev(2, [{iov_base="free(): invalid pointer", iov_len=23}, {iov_base="\n", iov_len=1}], 2free(): invalid pointer
) = 24

The text was updated successfully, but these errors were encountered:

ellieayla · 2021-05-17T16:04:39Z

I can reproduce this (and no k8s required) on a M1 MBP.

➜  ~ uname -a
Darwin Nemu.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101 arm64

Using either of these starting images:

image python:3.9.5-slim-buster

apt install strace
python3 -m pip install google-cloud-pubsub

image debian:bullseye-slim

apt install python3 python3-pip libsodium-dev wget python3-setuptools libpq-dev git python3-dev build-essential strace
python3 -m pip install google-cloud-pubsub

@hazcod observed a fault when importing google.cloud.pubsub_v1 as above;

# strace python3 -c 'from google.cloud import pubsub_v1'
...
writev(2, [{iov_base="free(): invalid pointer", iov_len=23}, {iov_base="\n", iov_len=1}], 2free(): invalid pointer
) = 24
...
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=306, si_uid=0} ---
+++ killed by SIGABRT +++
Aborted

But importing google.cloud.pubsub_v1 isn't really required here; the fault can be reproduced with importing only these two dependencies...

python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted

Importing either by itself doesn't crash. Further, importing them in the other order doesn't crash. I do not understand why.

python3 -c 'import grpc, proto.message; print(grpc, proto.message)'
<module 'grpc' from '/usr/local/lib/python3.9/site-packages/grpc/__init__.py'> <module 'proto.message' from '/usr/local/lib/python3.9/site-packages/proto/message.py'>

hazcod · 2021-05-17T21:13:00Z

Opened up https://bugs.python.org/issue44161

plamut · 2021-05-17T22:06:50Z

This sounds like an external issue in one of the dependencies. It's the first time I'm seeing this, thanks for the report!

@lidizheng Is this issue familiar? Can you reproduce it as reported?
(basically trying to narrow it down to either proto-plus or grpcio)

@hazcod Can you please also share the versions of the proto-plus and grpcio dependencies used in the check? Or better yet, the entire pip freeze output? Thanks!

lidizheng · 2021-05-17T22:16:16Z

Crashing depending on the import order of grpcio and protobuf has happened before: grpc/grpc#24897. The quick fix is try to upgrade both grpcio and protobuf to in-sync, it's grpcio==1.37.1 grpcio-tools=1.37.1 protobuf==3.15.2.

CC @gnossen who found the ABI incompatibility in the first place.

EDIT: can not reproduce with macOS 11.3.1 + 3.7.9, here is the pip freeze: https://gist.github.com/lidizheng/391d0eb5f8be52ced2d259c9f806ee82

gnossen · 2021-05-17T23:05:27Z

The quick fix is try to upgrade both grpcio and protobuf to in-sync,

Actually, this issue should be fixed, regardless of protobuf version.

@alanjcastonguay Some questions for you:

What version of grpcio do you have installed?
Does this issue still occur if you update to the latest version of grpcio? (1.37.1)
Do you have grpcio-tools installed as well? Or just grpcio? If both, please try upgrading grpcio-tools as well.
If you do have grpcio-tools installed, try removing it. Does the issue persist?

If upgrading doesn't fix the issue for you, please help me generate a symbolized core dump. Install grpcio (and grpcio-tools) using our nightly builds, which do not have symbols stripped.

gnossen · 2021-05-17T23:15:03Z

I just noticed that this is in Docker on Mac M1, not x86 Mac OS, which the fix was for. Can you confirm which artifacts you're downloading? Is it grpcio-1.37.1-cp39-cp39-manylinux2014_aarch64.whl? These aarch64 artifacts are brand new. If this is the problematic artifact, then it's likely that a very similar issue to the one fixed by grpc/grpc#24992 is at fault here.

ellieayla · 2021-05-17T23:18:34Z

Starting from python:3.9.5-slim-buster, which has no packages.

➜  ~ docker run -it python:3.9.5-slim-buster bash
root@9bd98952a250:/# python3 -m pip freeze
root@9bd98952a250:/# python3 -m pip install google-cloud-pubsub
Collecting google-cloud-pubsub
  Downloading google_cloud_pubsub-2.4.2-py2.py3-none-any.whl (212 kB)
     |████████████████████████████████| 212 kB 3.9 MB/s
Collecting proto-plus>=1.7.1
  Downloading proto_plus-1.18.1-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 1.8 MB/s
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Downloading grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
Collecting libcst>=0.3.10
  Downloading libcst-0.3.19-py3-none-any.whl (513 kB)
     |████████████████████████████████| 513 kB 18.7 MB/s
Collecting google-api-core[grpc]<2.0.0dev,>=1.22.2
  Downloading google_api_core-1.26.3-py2.py3-none-any.whl (93 kB)
     |████████████████████████████████| 93 kB 3.4 MB/s
Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.9/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-pubsub) (56.2.0)
Collecting six>=1.13.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting packaging>=14.3
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 12.9 MB/s
Collecting google-auth<2.0dev,>=1.21.1
  Downloading google_auth-1.30.0-py2.py3-none-any.whl (146 kB)
     |████████████████████████████████| 146 kB 37.5 MB/s
Collecting pytz
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 22.2 MB/s
Collecting requests<3.0.0dev,>=2.18.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 10.0 MB/s
Collecting protobuf>=3.12.0
  Downloading protobuf-3.17.0-cp39-cp39-manylinux2014_aarch64.whl (925 kB)
     |████████████████████████████████| 925 kB 11.1 MB/s
Collecting googleapis-common-protos<2.0dev,>=1.6.0
  Downloading googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
     |████████████████████████████████| 198 kB 11.8 MB/s
Collecting grpcio<2.0dev,>=1.29.0
  Downloading grpcio-1.37.1-cp39-cp39-manylinux2014_aarch64.whl (37.1 MB)
     |████████████████████████████████| 37.1 MB 27.7 MB/s
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 68.1 MB/s
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting typing-inspect>=0.4.0
  Downloading typing_inspect-0.6.0-py3-none-any.whl (8.1 kB)
Collecting typing-extensions>=3.7.4.2
  Downloading typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Collecting pyyaml>=5.2
  Downloading PyYAML-5.4.1-cp39-cp39-manylinux2014_aarch64.whl (788 kB)
     |████████████████████████████████| 788 kB 18.9 MB/s
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 11.1 MB/s
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 10.2 MB/s
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
     |████████████████████████████████| 153 kB 29.2 MB/s
Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 25.6 MB/s
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 17.5 MB/s
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 15.4 MB/s
Collecting mypy-extensions>=0.3.0
  Downloading mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Building wheels for collected packages: grpc-google-iam-v1
  Building wheel for grpc-google-iam-v1 (setup.py) ... done
  Created wheel for grpc-google-iam-v1: filename=grpc_google_iam_v1-0.12.3-py3-none-any.whl size=18499 sha256=63c626f89627ac51175131d58db3a10351b17b3028f9c0697b286fa919b8a012
  Stored in directory: /root/.cache/pip/wheels/a1/b4/ad/f2f4118c678bbc6496d5b7057d2508ba51f66009700b1afe8a
Successfully built grpc-google-iam-v1
Installing collected packages: six, pyasn1, urllib3, rsa, pyparsing, pyasn1-modules, protobuf, idna, chardet, certifi, cachetools, typing-extensions, requests, pytz, packaging, mypy-extensions, grpcio, googleapis-common-protos, google-auth, typing-inspect, pyyaml, google-api-core, proto-plus, libcst, grpc-google-iam-v1, google-cloud-pubsub
Successfully installed cachetools-4.2.2 certifi-2020.12.5 chardet-4.0.0 google-api-core-1.26.3 google-auth-1.30.0 google-cloud-pubsub-2.4.2 googleapis-common-protos-1.53.0 grpc-google-iam-v1-0.12.3 grpcio-1.37.1 idna-2.10 libcst-0.3.19 mypy-extensions-0.4.3 packaging-20.9 proto-plus-1.18.1 protobuf-3.17.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-2.4.7 pytz-2021.1 pyyaml-5.4.1 requests-2.25.1 rsa-4.7.2 six-1.16.0 typing-extensions-3.10.0.0 typing-inspect-0.6.0 urllib3-1.26.4
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

So installing google_cloud_pubsub==2.4.2 pulled in these packages:

root@9bd98952a250:/# python3 -m pip freeze
cachetools==4.2.2
certifi==2020.12.5
chardet==4.0.0
google-api-core==1.26.3
google-auth==1.30.0
google-cloud-pubsub==2.4.2
googleapis-common-protos==1.53.0
grpc-google-iam-v1==0.12.3
grpcio==1.37.1
idna==2.10
libcst==0.3.19
mypy-extensions==0.4.3
packaging==20.9
proto-plus==1.18.1
protobuf==3.17.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
pytz==2021.1
PyYAML==5.4.1
requests==2.25.1
rsa==4.7.2
six==1.16.0
typing-extensions==3.10.0.0
typing-inspect==0.6.0
urllib3==1.26.4

The simple-case reproduction from above:

root@9bd98952a250:/# python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted

The questions posed:

What version of grpcio do you have installed? grpcio==1.37.1
Does this issue still occur if you update to the latest version of grpcio? (1.37.1) Yes
Do you have grpcio-tools installed as well? Or just grpcio? If both, please try upgrading grpcio-tools as well. Only grpcio, no grpcio-tools.
If you do have grpcio-tools installed, try removing it. Does the issue persist? Yes, because that's the initial case.
Can you confirm which artifacts you're downloading? Is it grpcio-1.37.1-cp39-cp39-manylinux2014_aarch64.whl? Yes.

Questions not posed:

If you do not have grpcio-tools installed, try installing it. Does the issue persist? Yes.

root@9bd98952a250:/# python3 -m pip install grpcio-tools
Collecting grpcio-tools
  Downloading grpcio_tools-1.37.1-cp39-cp39-manylinux2014_aarch64.whl (25.5 MB)
     |████████████████████████████████| 25.5 MB 47 kB/s
Requirement already satisfied: protobuf<4.0dev,>=3.5.0.post1 in /usr/local/lib/python3.9/site-packages (from grpcio-tools) (3.17.0)
Requirement already satisfied: grpcio>=1.37.1 in /usr/local/lib/python3.9/site-packages (from grpcio-tools) (1.37.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from grpcio-tools) (56.2.0)
Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.9/site-packages (from grpcio>=1.37.1->grpcio-tools) (1.16.0)
Installing collected packages: grpcio-tools
Successfully installed grpcio-tools-1.37.1

root@9bd98952a250:/# python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted

gnossen · 2021-05-17T23:33:07Z

CC @jtattermusch who built both the aarch64 grpcio wheels and the aarch64 protobuf wheels.

ellieayla · 2021-05-17T23:35:03Z

If upgrading doesn't fix the issue for you, please help me generate a symbolized core dump. Install grpcio (and grpcio-tools) using our nightly builds, which do not have symbols stripped.

Using only grpcio-1.39.0 + grpcio_tools-1.39.0 + proto_plus-1.18.1 (+ six-1.16.0 + protobuf-3.17.0, which are pulled in as dependencies).

➜  ~ docker run -it python:3.9.5-slim-buster bash
root@fa796b3a1f02:/# python3 -m pip install https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio/grpcio-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio-tools/grpcio_tools-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl proto_plus==1.18.1
Collecting grpcio==1.39.0.dev0
  Downloading https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio/grpcio-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl (39.0 MB)
     |████████████████████████████████| 39.0 MB 92.2 MB/s
Collecting grpcio-tools==1.39.0.dev0
  Downloading https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio-tools/grpcio_tools-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl (25.5 MB)
     |████████████████████████████████| 25.5 MB 41.0 MB/s
Collecting proto_plus==1.18.1
  Downloading proto_plus-1.18.1-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 862 kB/s
Collecting six>=1.5.2
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from grpcio-tools==1.39.0.dev0) (56.2.0)
Collecting protobuf<4.0dev,>=3.5.0.post1
  Downloading protobuf-3.17.0-cp39-cp39-manylinux2014_aarch64.whl (925 kB)
     |████████████████████████████████| 925 kB 6.2 MB/s
Installing collected packages: six, protobuf, grpcio, proto-plus, grpcio-tools
Successfully installed grpcio-1.39.0.dev0 grpcio-tools-1.39.0.dev0 proto-plus-1.18.1 protobuf-3.17.0 six-1.16.0
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

root@fa796b3a1f02:/# ulimit -c unlimited

root@fa796b3a1f02:/# python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted (core dumped)

root@fa796b3a1f02:/# python3 -c 'import grpc, proto.message'
# no crash

root@fa796b3a1f02:/# python3 -m pip freeze
grpcio @ https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio/grpcio-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl
grpcio-tools @ https://packages.grpc.io/archive/2021/05/cc9326c93650f5e853fe02922ddb476af70ce303-650eaf9b-0217-4f48-853e-67c78ec48cb7/python/grpcio-tools/grpcio_tools-1.39.0.dev0-cp39-cp39-manylinux2014_aarch64.whl
proto-plus==1.18.1
protobuf==3.17.0
six==1.16.0

Attached: py39-googleapis-python-pubsub-issue-414.core.gz decompressing to give

➜  ~ shasum py39-googleapis-python-pubsub-issue-414.core
26975b5d6e3a7f57e2a410ad51976b27d7b32851  py39-googleapis-python-pubsub-issue-414.core
➜  ~ file py39-googleapis-python-pubsub-issue-414.core
py39-googleapis-python-pubsub-issue-414.core: ELF 64-bit LSB core file, ARM aarch64, version 1 (SYSV), SVR4-style, from 'python3 -c import proto.message, grpc;'

gnossen · 2021-05-17T23:49:08Z

Either my gdb-multiarch isn't set up properly, or there's something very wrong in this process:

Core was generated by `python3 -c import proto.message, grpc;'.
Program terminated with signal SIGABRT, Aborted.
#0  0x0000ffff88ba6714 in ?? ()
(gdb) bt
#0  0x0000ffff88ba6714 in ?? ()
#1  0x0000ffff89217010 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) disassemble 0x0000ffff88ba6700,0x0000ffff88ba6730
Dump of assembler code from 0xffff88ba6700 to 0xffff88ba6730:
   0x0000ffff88ba6700:  udf     #0
   0x0000ffff88ba6704:  udf     #0
   0x0000ffff88ba6708:  udf     #0
   0x0000ffff88ba670c:  udf     #0
   0x0000ffff88ba6710:  udf     #0
=> 0x0000ffff88ba6714:  udf     #0
   0x0000ffff88ba6718:  udf     #0
   0x0000ffff88ba671c:  udf     #0
   0x0000ffff88ba6720:  udf     #0
   0x0000ffff88ba6724:  udf     #0
   0x0000ffff88ba6728:  udf     #0
   0x0000ffff88ba672c:  udf     #0
End of assembler dump.
(gdb) f 1
#1  0x0000ffff89217010 in ?? ()
(gdb) disassemble 0x0000ffff89217000,0x0000ffff89217020
Dump of assembler code from 0xffff89217000 to 0xffff89217020:
   0x0000ffff89217000:  udf     #0
   0x0000ffff89217004:  udf     #0
   0x0000ffff89217008:  .inst   0x89217000 ; undefined
   0x0000ffff8921700c:  udf     #65535
=> 0x0000ffff89217010:  udf     #0
   0x0000ffff89217014:  udf     #0
   0x0000ffff89217018:  udf     #0
   0x0000ffff8921701c:  udf     #0
End of assembler dump.

ellieayla · 2021-05-18T00:01:48Z

I can install gdb inside that same container and grab a backtrace, though I've no idea how to get the inner python bits or what you might be looking for.

root@fa796b3a1f02:/# gdb /usr/local/bin/python3 /core
...
Reading symbols from /usr/local/bin/python3...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 15]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `python3 -c import proto.message, grpc;'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000ffff88b948e8 in __GI_abort () at abort.c:79
#2  0x0000ffff88be06f8 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffff88ca0bd8 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x0000ffff88be6cb4 in malloc_printerr (str=str@entry=0xffff88c9c5d8 "free(): invalid pointer") at malloc.c:5341
#4  0x0000ffff88be856c in _int_free (av=0xffff88ce0a98 <main_arena>, p=0xffff87fd2950 <(anonymous namespace)::ctype_c+560>, have_lock=0)
    at malloc.c:4165
#5  0x0000ffff87e5a948 in std::locale::_Impl::_M_install_facet (this=0xffff87fd2248 <(anonymous namespace)::c_locale_impl>, __idp=<optimized out>,
    __fp=0xffff87fd22e0 <(anonymous namespace)::numpunct_c>) at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale.cc:352
#6  0x0000ffff87e45df0 in std::locale::_Impl::_M_init_facet<std::numpunct<char> > (__facet=0xffff87fd22e0 <(anonymous namespace)::numpunct_c>,
    this=0xffff87fd2248 <(anonymous namespace)::c_locale_impl>)
    at /dockcross/crosstool/toolchain/.build/aarch64-unknown-linux-gnueabi/build/build-cc-gcc-final/aarch64-unknown-linux-gnueabi/libstdc++-v3/include/bits/locale_classes.h:564
#7  std::locale::_Impl::_Impl (this=0xffff87fd2248 <(anonymous namespace)::c_locale_impl>, __refs=<optimized out>)
    at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale_init.cc:402
#8  0x0000ffff87e46710 in std::locale::_S_initialize_once ()
    at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale_init.cc:267
#9  0x0000ffff88dda250 in __pthread_once_slow (once_control=0xffff87fd3668 <std::locale::_S_once>,
    init_routine=0xffff87e466f0 <std::locale::_S_initialize_once()>) at pthread_once.c:116
#10 0x0000ffff87e4675c in __gthread_once (__once=<optimized out>, __func=0xffff87e466f0 <std::locale::_S_initialize_once()>)
    at /dockcross/crosstool/toolchain/.build/aarch64-unknown-linux-gnueabi/build/build-cc-gcc-final/aarch64-unknown-linux-gnueabi/libstdc++-v3/include/aarch64-unknown-linux-gnueabi/bits/gthr-default.h:699
#11 std::locale::_S_initialize () at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale_init.cc:276
#12 0x0000ffff87e46794 in std::locale::locale (this=0xffff87fe63c0 <__gnu_internal::buf_cout_sync+56>)
    at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale_init.cc:210
#13 0x0000ffff87e60510 in __gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >::stdio_sync_filebuf (
    this=0xffff87fe6388 <__gnu_internal::buf_cout_sync>, __f=0xffff87f70a30)
    at /dockcross/crosstool/toolchain/.build/aarch64-unknown-linux-gnueabi/build/build-cc-gcc-final/aarch64-unknown-linux-gnueabi/libstdc++-v3/include/ext/stdio_sync_filebuf.h:77
#14 0x0000ffff87e59b24 in std::ios_base::Init::Init (this=<optimized out>)
    at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/ios_init.cc:85
#15 0x0000ffff879273e0 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535)
--Type <RET> for more, q to quit, c to continue without paging--
    at /usr/xcc/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/include/c++/4.9.4/iostream:74
#16 _GLOBAL__sub_I_grpc_context.cc(void) () at src/core/ext/filters/census/grpc_context.cc:38
#17 0x0000ffff8920093c in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0xfffffa0a87d8, env=0xfffffa0a87f8) at dl-init.c:72
#18 0x0000ffff89200a40 in call_init (env=0xfffffa0a87f8, argv=0xfffffa0a87d8, argc=3, l=<optimized out>) at dl-init.c:30
#19 _dl_init (main_map=main_map@entry=0xaaaac9a97900, argc=3, argv=0xfffffa0a87d8, env=0xfffffa0a87f8) at dl-init.c:119
#20 0x0000ffff89204750 in dl_open_worker (a=a@entry=0xfffffa0a5140) at dl-open.c:517
#21 0x0000ffff88c7de58 in __GI__dl_catch_exception (exception=exception@entry=0xfffffa0a5128, operate=operate@entry=0xffff89204430 <dl_open_worker>,
    args=args@entry=0xfffffa0a5140) at dl-error-skeleton.c:196
#22 0x0000ffff89204098 in _dl_open (file=0xffff88037c50 "/usr/local/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-aarch64-linux-gnu.so",
    mode=-2147483646, caller_dlopen=0xffff89071de0, nsid=-2, argc=3, argv=<optimized out>, env=<optimized out>) at dl-open.c:599
#23 0x0000ffff88db8094 in dlopen_doit (a=a@entry=0xfffffa0a53f8) at dlopen.c:66
#24 0x0000ffff88c7de58 in __GI__dl_catch_exception (exception=exception@entry=0xfffffa0a5370, operate=operate@entry=0xffff88db8030 <dlopen_doit>,
    args=args@entry=0xfffffa0a53f8) at dl-error-skeleton.c:196
#25 0x0000ffff88c7df00 in __GI__dl_catch_error (objname=objname@entry=0xaaaac98db5b0, errstring=errstring@entry=0xaaaac98db5b8,
    mallocedp=mallocedp@entry=0xaaaac98db5a8, operate=operate@entry=0xffff88db8030 <dlopen_doit>, args=args@entry=0xfffffa0a53f8)
    at dl-error-skeleton.c:215
#26 0x0000ffff88db87e0 in _dlerror_run (operate=operate@entry=0xffff88db8030 <dlopen_doit>, args=args@entry=0xfffffa0a53f8) at dlerror.c:163
#27 0x0000ffff88db8134 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#28 0x0000ffff89071de0 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#29 0x0000ffff89060104 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#30 0x0000ffff8905fd60 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#31 0x0000ffff88f5bd54 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#32 0x0000ffff88f2a138 in PyVectorcall_Call () from /usr/local/lib/libpython3.9.so.1.0
#33 0x0000ffff88fae9dc in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#34 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#35 0x0000ffff88f29224 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#36 0x0000ffff88fb08d0 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#37 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#38 0x0000ffff88facfac in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#39 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#40 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
--Type <RET> for more, q to quit, c to continue without paging--
#41 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#42 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#43 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#44 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#45 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#46 0x0000ffff88f28c08 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#47 0x0000ffff88f2a488 in _PyObject_CallMethodIdObjArgs () from /usr/local/lib/libpython3.9.so.1.0
#48 0x0000ffff88fcd0f4 in PyImport_ImportModuleLevelObject () from /usr/local/lib/libpython3.9.so.1.0
#49 0x0000ffff88fa9674 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#50 0x0000ffff88f5be88 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#51 0x0000ffff88f29ff0 in _PyObject_Call () from /usr/local/lib/libpython3.9.so.1.0
#52 0x0000ffff88fae9dc in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#53 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#54 0x0000ffff88f29224 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#55 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#56 0x0000ffff88fabe98 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#57 0x0000ffff88f29224 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#58 0x0000ffff88f28c08 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#59 0x0000ffff88f2a488 in _PyObject_CallMethodIdObjArgs () from /usr/local/lib/libpython3.9.so.1.0
#60 0x0000ffff88fcd1d4 in PyImport_ImportModuleLevelObject () from /usr/local/lib/libpython3.9.so.1.0
#61 0x0000ffff88fb057c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#62 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#63 0x0000ffff88fab69c in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.9.so.1.0
#64 0x0000ffff88fab628 in PyEval_EvalCodeEx () from /usr/local/lib/libpython3.9.so.1.0
#65 0x0000ffff88fab5d4 in PyEval_EvalCode () from /usr/local/lib/libpython3.9.so.1.0
#66 0x0000ffff88fa9254 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#67 0x0000ffff88f5bd54 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#68 0x0000ffff88f2a138 in PyVectorcall_Call () from /usr/local/lib/libpython3.9.so.1.0
#69 0x0000ffff88fae9dc in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#70 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#71 0x0000ffff88f29224 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
--Type <RET> for more, q to quit, c to continue without paging--
#72 0x0000ffff88fb08d0 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#73 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#74 0x0000ffff88facfac in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#75 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#76 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#77 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#78 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#79 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#80 0x0000ffff88f28c08 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#81 0x0000ffff88f2a488 in _PyObject_CallMethodIdObjArgs () from /usr/local/lib/libpython3.9.so.1.0
#82 0x0000ffff88fcd0f4 in PyImport_ImportModuleLevelObject () from /usr/local/lib/libpython3.9.so.1.0
#83 0x0000ffff88fb057c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#84 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#85 0x0000ffff88fab69c in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.9.so.1.0
#86 0x0000ffff88fab628 in PyEval_EvalCodeEx () from /usr/local/lib/libpython3.9.so.1.0
#87 0x0000ffff88fab5d4 in PyEval_EvalCode () from /usr/local/lib/libpython3.9.so.1.0
#88 0x0000ffff8906b1c8 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#89 0x0000ffff8906b10c in ?? () from /usr/local/lib/libpython3.9.so.1.0
#90 0x0000ffff8906b05c in PyRun_StringFlags () from /usr/local/lib/libpython3.9.so.1.0
#91 0x0000ffff8906af78 in PyRun_SimpleStringFlags () from /usr/local/lib/libpython3.9.so.1.0
#92 0x0000ffff89073fc4 in Py_RunMain () from /usr/local/lib/libpython3.9.so.1.0
#93 0x0000ffff89073b84 in Py_BytesMain () from /usr/local/lib/libpython3.9.so.1.0
#94 0x0000ffff88b94d24 in __libc_start_main (main=0xaaaabe7e08c0, argc=3, argv=0xfffffa0a87d8, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#95 0x0000aaaabe7e08f8 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

gnossen · 2021-05-18T00:24:08Z

I've no idea how to get the inner python bits or what you might be looking for.

Yeah, I need to figure out what's wrong with my gdb setup to effectively debug this, but what you just posted is super helpful on it's own. This line in particular is telling:

#15 0x0000ffff879273e0 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535)

It looks like the problem has to do with static variable initialization/destruction.

@jtattermusch is probably better equipped to reproduce/debug at this point.

plamut · 2021-05-18T05:17:16Z

@gnossen Thanks for jumping in and the research so far!

Should we transfer this issue to grpcio, as it's not directly related to PubSub, the latter just uses grpcio as a dependency?

OTOH it would still be good to keep it here for visibility, and if you guys don't mind tracking it here, I'm also fine with it.

hazcod · 2021-05-18T06:05:24Z

Thank you all for co-debugging! I've opened up grpc/grpc#26279 to the best of my knowledge.
Please chime in there in case I missed something.

hazcod · 2021-05-19T12:07:43Z

Curiously, is there a workaround for now to use python-pubsub on M1?
Freezing pip to: (?)

grpcio==1.37.1 grpcio-tools=1.37.1 protobuf==3.15.2

ellieayla · 2021-05-19T12:43:16Z

One workaround appears to be: import in this order:

import grpc
import proto.message
from google.cloud import pubsub_v1

jtattermusch · 2021-06-08T05:26:06Z

The issue has now been fixed in grpc/grpc#26279.
The will be in the next grpc-python release (1.39.x) and in the next patch release of 1.38.x.

plamut · 2021-06-08T07:25:58Z

@jtattermusch This sounds great, thanks!

plamut · 2021-06-28T15:10:59Z

This has been fixed and release in grpcio 1.38.1. Will update the version pin.

product-auto-label bot added the api: pubsub Issues related to the googleapis/python-pubsub API. label May 17, 2021

hazcod mentioned this issue May 17, 2021

Kubernetes on Docker for Mac: free(): invalid pointer docker/for-mac#5694

Closed

3 tasks

plamut added the external This issue is blocked on a bug with the actual product. label May 17, 2021

plamut added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. labels May 17, 2021

hazcod mentioned this issue May 18, 2021

free(): invalid pointer crash on specific import order (on ARM64 linux) grpc/grpc#26279

Closed

plamut mentioned this issue Jun 8, 2021

free(): invalid pointer Aborted error when importing from google.cloud import bigquery in python googleapis/python-bigquery#695

Closed

plamut self-assigned this Jun 28, 2021

This was referenced Jun 28, 2021

fix: require grpcio >= 1.38.1 to prevent a possible crash googleapis/python-bigquery#725

Merged

fix: require grpcio >= 1.38.1 to prevent a possible crash #435

Merged

plamut closed this as completed in #435 Jul 1, 2021

julian-risch mentioned this issue Sep 20, 2021

Haystack REST API not starting deepset-ai/haystack#1443

Closed

MattDietz mentioned this issue Aug 22, 2022

AttributeError: 'NoneType' object has no attribute 'message_types_by_name' protocolbuffers/protobuf#10075

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crash: free(): invalid pointer #414

crash: free(): invalid pointer #414

hazcod commented May 17, 2021 •

edited

ellieayla commented May 17, 2021 •

edited

hazcod commented May 17, 2021

plamut commented May 17, 2021

lidizheng commented May 17, 2021 •

edited

gnossen commented May 17, 2021

gnossen commented May 17, 2021

ellieayla commented May 17, 2021

gnossen commented May 17, 2021

ellieayla commented May 17, 2021 •

edited

gnossen commented May 17, 2021

ellieayla commented May 18, 2021

gnossen commented May 18, 2021

plamut commented May 18, 2021 •

edited

hazcod commented May 18, 2021

hazcod commented May 19, 2021 •

edited

ellieayla commented May 19, 2021 •

edited

jtattermusch commented Jun 8, 2021

plamut commented Jun 8, 2021

plamut commented Jun 28, 2021

crash: free(): invalid pointer #414

crash: free(): invalid pointer #414

Comments

hazcod commented May 17, 2021 • edited

Environment details

Steps to reproduce

Code example

Stack trace

ellieayla commented May 17, 2021 • edited

hazcod commented May 17, 2021

plamut commented May 17, 2021

lidizheng commented May 17, 2021 • edited

gnossen commented May 17, 2021

gnossen commented May 17, 2021

ellieayla commented May 17, 2021

gnossen commented May 17, 2021

ellieayla commented May 17, 2021 • edited

gnossen commented May 17, 2021

ellieayla commented May 18, 2021

gnossen commented May 18, 2021

plamut commented May 18, 2021 • edited

hazcod commented May 18, 2021

hazcod commented May 19, 2021 • edited

ellieayla commented May 19, 2021 • edited

jtattermusch commented Jun 8, 2021

plamut commented Jun 8, 2021

plamut commented Jun 28, 2021

hazcod commented May 17, 2021 •

edited

ellieayla commented May 17, 2021 •

edited

lidizheng commented May 17, 2021 •

edited

ellieayla commented May 17, 2021 •

edited

plamut commented May 18, 2021 •

edited

hazcod commented May 19, 2021 •

edited

ellieayla commented May 19, 2021 •

edited