New Relic Instrumentor on FastAPI app fails when both OpenTelemetry and Middleware are setup

[csuriano23]

Instrumenting a FastAPI app with New Relic executable (newrelic-admin run-program) causes failures in app endpoints due to a type mismatch on the ASGI middlewares chain

Description
Using this minimal example FastAPI app:

import newrelic.agent
from fastapi import FastAPI
from fastapi.middleware.gzip import GZipMiddleware
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.trace import TracerProvider

from .models.responses import JSONResponse

app = FastAPI(default_response_class=JSONResponse)

app.add_middleware(GZipMiddleware, minimum_size=1000)


@app.get("/healthcheck")
async def healthcheck() -> JSONResponse:
    newrelic.agent.ignore_transaction()
    return JSONResponse({})


# Setup OpenTelemetry
tracer = TracerProvider()
trace.set_tracer_provider(tracer)
FastAPIInstrumentor.instrument_app(app, excluded_urls="/docs,/healthcheck/")

Dependencies versions (excerpt from pyproject.toml):

[tool.poetry.dependencies]
python = "^3.10"
newrelic = "9.9.0"
uvicorn = ">=0.21.1,<1"  # installed 0.29.0
fastapi = ">=0.109.2,<1"  # installed FastAPI 0.110.1 with Starlette 0.37.2
opentelemetry-instrumentation-fastapi = "^0.45b0"
opentelemetry-instrumentation-requests = "^0.45b0"
opentelemetry-instrumentation-pymongo = "^0.45b0"
opentelemetry-instrumentation-redis = "^0.45b0"
opentelemetry-instrumentation-logging = "^0.45b0"
opentelemetry-api = "^1.24.0"
opentelemetry-sdk = "^1.24.0"

Running the app with the command:

newrelic-admin run-program bash -c "uvicorn main:app --host 0.0.0.0 --port 8000"

causes an exception when calling the app endpoints (in the example: "/healthcheck")

The exception stacktrace is:

/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py:-1: RuntimeWarning: coroutine 'middleware_wrapper' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
INFO:     127.0.0.6:59175 - "GET /healthcheck/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/newrelic/api/asgi_application.py", line 357, in nr_async_asgi
    return await coro
  File "/usr/local/lib/python3.10/site-packages/newrelic/common/async_proxy.py", line 148, in __next__
    return self.send(None)
  File "/usr/local/lib/python3.10/site-packages/newrelic/common/async_proxy.py", line 120, in send
    return self.__wrapped__.send(value)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/newrelic/api/asgi_application.py", line 357, in nr_async_asgi
    return await coro
  File "/usr/local/lib/python3.10/site-packages/newrelic/common/async_proxy.py", line 148, in __next__
    return self.send(None)
  File "/usr/local/lib/python3.10/site-packages/newrelic/common/async_proxy.py", line 120, in send
    return self.__wrapped__.send(value)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "<string>", line 5, in wrapper
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/newrelic/hooks/framework_starlette.py", line 112, in middleware_wrapper
    return await FunctionTraceWrapper(wrapped, name=name)(*args, **kwargs)
  File "<string>", line 5, in wrapper
  File "/usr/local/lib/python3.10/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 575, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/asgiref/compatibility.py", line 35, in new_application
    return await instance(receive, send)
TypeError: 'coroutine' object is not callable

If I remove:

• the GZIP middleware, but NOT the OpenTelemetry instrumentor

OR

• the OpenTelemetry instrumentor, but NOT the GZIP middleware

this is not happening.
The OpenTelemetry instrumentor itself contains an ASGI middleware, so I think something is not working properly in the interceptors chain when both are installed. Anyway the OpenTelemetry instrumentor shouldn't be incompatible with the NewRelic one.