Asynchronous context manager crash with langchain-openai >= 0.1.2 #9077
Labels
Comments
|
Adding another stacktrace (not using async) |
|
|
5 tasks
|
Will a fix for this be included in v2.9.0? |
|
Tagging @Yun-Kim for visibility since I see you in most of openai related commits Please retag if you know who to ask |
|
Thanks for raising the issue, and apologies for the late response. We're working on getting a fix out ASAP. In the meantime as a workaround, I recommend pinning your version of |
|
Confirming this is happening for |
Yun-Kim
added a commit
that referenced
this issue
May 21, 2024
Fixes #9077. This PR changes the way we patch streamed completion/chat responses in the OpenAI integration. Previously we had been returning generator functions that yielded the value of each chunk. However, this was incorrect as `OpenAIStream/OpenAIAsyncStream` objects can be used as context managers as of `openai>=1.6.0`, which means returning a generator function to replace `OpenAIStream` would break user applications that tried to access stream responses as context managers: ```python with openai.OpenAI().completions.create(..., stream=True) as response: for chunk in response: print(chunk) ``` There is no introduced/changed functionality other than the fact that we now returned a wrapped `TracedOpenAIStream` object instead of the previous traced generator function. Note: This PR may seem much larger than it really is, we also updated the tested versions of OpenAI so the riot requirement lockfiles have been changed. ## Checklist - [x] Change(s) are motivated and described in the PR description - [x] Testing strategy is described if automated tests are not included in the PR - [x] Risks are described (performance impact, potential for breakage, maintainability) - [x] Change is maintainable (easy to change, telemetry, documentation) - [x] [Library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) are followed or label `changelog/no-changelog` is set - [x] Documentation is included (in-code, generated user docs, [public corp docs](https://github.com/DataDog/documentation/)) - [x] Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) - [x] If this PR changes the public interface, I've notified `@DataDog/apm-tees`. ## Reviewer Checklist - [x] Title is accurate - [x] All changes are related to the pull request's stated goal - [x] Description motivates each change - [x] Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - [x] Testing strategy adequately addresses listed risks - [x] Change is maintainable (easy to change, telemetry, documentation) - [x] Release note makes sense to a user of the library - [x] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - [x] Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
|
Thanks for getting a fix for this @Yun-Kim! Just to point out, I ran into this with langchain, but it is an issue with just openai and ddtrace: The following test illustrates the issue using openai without langchain: import openai
import pytest
@pytest.mark.asyncio
async def test_openai_client():
client = openai.AsyncOpenAI().chat.completions
# this exhibits the issue only when run under ddtrace
# TypeError: 'async_generator' object does not support the asynchronous context manager protocol
response = await client.create(
messages=[
{
"role": "user",
"content": "what is the meaning of life?",
}
],
model="gpt-3.5-turbo-0125",
stream=True,
)
# type of response:
# without ddtrace: <class 'openai.AsyncStream'>
# with ddtrace: <class 'async_generator'>
print(type(response))
async with response as resp:
async for chunk in resp:
print(chunk)Failure output: |
|
Yes, thanks for clarifying @mattflo, this is a bug with our openAI integration which didn't account for the fact that OpenAI started making |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary of problem
ddtracebreaks streaming withlangchain-openai >= 0.1.2Which version of dd-trace-py are you using?
2.8.2
Which version of pip are you using?
23.3.1
How can we reproduce your problem?
The following code will crash:
With the following exception:
The root cause appears to be in the way
ChatOpenAIis patched. The langchain-openai code was recently updated to use context managers:(PR: https://github.com/langchain-ai/langchain/pull/18013/files#diff-197e34e7a297cef90049a95d035073870fb484896bab67f800a95fe190ab02bfR560)
However, the tracer returns a type that isn't compatible
The text was updated successfully, but these errors were encountered: