Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OtelBatchSpanProcessor threading KeyError #3884

Open
frx08 opened this issue Apr 26, 2024 · 2 comments
Open

OtelBatchSpanProcessor threading KeyError #3884

frx08 opened this issue Apr 26, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@frx08
Copy link

frx08 commented Apr 26, 2024

Environment:

ubuntu:22.04
python:3.10

libs installed:

...
opentelemetry-api==1.24.0
opentelemetry-sdk==1.24.0
opentelemetry-exporter-gcp-trace==1.6.0
...

app.py

from opentelemetry import trace
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

tracer_provider = TracerProvider()
trace.set_tracer_provider(tracer_provider)
gcp_exporter = CloudTraceSpanExporter()
span_processor = BatchSpanProcessor(gcp_exporter)
tracer_provider.add_span_processor(span_processor)

running with:
gunicorn --worker-class geventwebsocket.gunicorn.workers.GeventWebSocketWorker -w 1 --threads 20 app:app

I'm running my service on gcloud app engine with this configuration but many times when I create a new span I get the following exception and the service is restarted (i'm using supervisord)

Traceback (most recent call last):    File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run    File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.10/threading.py", line 1008, in _bootstrap_inner
    del _limbo[self]
KeyError: <Thread(OtelBatchSpanProcessor, stopped daemon 139781863898496)>
2024-04-26T18:20:34Z <Greenlet at 0x7f218057fd80: <bound method Thread._bootstrap of <Thread(OtelBatchSpanProcessor, stopped daemon 139781863898496)>>> failed with KeyError
[2024-04-26 18:20:35 +0000] [8] [ERROR] Worker (pid:12) was sent code 139!
2024-04-26 18:20:35,177 INFO reaped unknown pid 164 (exit status 0)
2024-04-26 18:20:35,179 INFO reaped unknown pid 147 (exit status 0)
2024-04-26 18:20:35,179 INFO reaped unknown pid 173 (exit status 0)

do you know how I can address this issue? As you can see I'm using gevent==23.9.1

@frx08 frx08 added the bug Something isn't working label Apr 26, 2024
@xrmx
Copy link
Contributor

xrmx commented Apr 29, 2024

If you use gevent you should monkey patch everything you use, and I don't see it from your code

@frx08
Copy link
Author

frx08 commented Apr 30, 2024

sorry I didn't include it, yes i'm patching everything at the very start of my script

from gevent import monkey
monkey.patch_all()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants