This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
The memory usage piles up over the time and leads to OOM #1624
Comments
|
No @Riki-1mg |
@app.get("/")
def read_root():
return {"Hello": "World"}Surely the expected behaviour here is to return If you want this function to return @app.get("/")
def read_root():
return {"Hello": "Sara"}Further, I can't reproduce your error on my "machine" (it's sitting on a cloud somewhere). You can see the full details here but everything looks to be working fine. I suspect that this is specific to your operating system setup, etc. Would you please provide some more info needed/useful to know in context of how to reproduce the error?
|
|
@teymour-aldridge : here is the debug statistics: /usr/local/lib/python3.6/site-packages/starlette/applications.py:136: size=1288 KiB (+1173 KiB), count=2290 (+2086), average=576 B KB /usr/local/lib/python3.6/threading.py:347: size=943 KiB (+854 KiB), count=1836 (+1657), average=526 B KB /usr/local/lib/python3.6/queue.py:145: size=919 KiB (+835 KiB), count=1783 (+1619), average=528 B KB /usr/local/lib/python3.6/asyncio/locks.py:233: size=885 KiB (+807 KiB), count=9633 (+8771), average=94 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:82: size=788 KiB (+717 KiB), count=6876 (+6264), average=117 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:77: size=751 KiB (+684 KiB), count=2289 (+2086), average=336 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:146: size=725 KiB (+662 KiB), count=15984 (+14611), average=46 B KB /usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py:376: size=657 KiB (+590 KiB), count=10490 (+9426), average=64 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:285: size=609 KiB (+555 KiB), count=4589 (+4183), average=136 B KB Scanned Lines that consumes more memory loop.run_until_complete(self.serve(sockets=sockets)) scope["app"] = self waiters_to_notify = _deque(_islice(all_waiters, n)) self.not_empty.notify() self._waiters = collections.deque() self.parser = httptools.HttpRequestParser(self) self.config = config self.parser.feed_data(data) for obj_elem in elem[4] self.timeout_keep_alive, self.timeout_keep_alive_handler |
|
@teymour-aldridge the above grows and cause OOM after a while |
|
@prav2019 I can't reproduce the bug on either my machine or on a cloud-hosted linux container; this leads me to believe that the problem is in the way that your machine/environment setup. In the issue template, it asks for the following fields – would you mind filling them in? Also, there's a "checklist" at the top of the issue which you should fill out! |
|
@teymour-aldridge , this usually happens when there are some traffic over a period of time. Usually this is happening in our prod environment. |
|
@teymour-aldridge , current status: /usr/local/lib/python3.6/site-packages/starlette/applications.py:136: size=2851 KiB (+1504 KiB), count=5069 (+2673), average=576 B KB /usr/local/lib/python3.6/threading.py:347: size=2104 KiB (+1110 KiB), count=4083 (+2155), average=528 B KB /usr/local/lib/python3.6/queue.py:145: size=2049 KiB (+1087 KiB), count=3974 (+2109), average=528 B KB /usr/local/lib/python3.6/asyncio/locks.py:233: size=1948 KiB (+1017 KiB), count=21295 (+11208), average=94 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:82: size=1744 KiB (+920 KiB), count=15226 (+8037), average=117 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:77: size=1663 KiB (+877 KiB), count=5068 (+2673), average=336 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:146: size=1598 KiB (+839 KiB), count=35181 (+18488), average=47 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:285: size=1349 KiB (+712 KiB), count=10163 (+5364), average=136 B KB /usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:216: size=1288 KiB (+676 KiB), count=30112 (+15818), average=44 B KB |
|
0.20.0 is a somewhat old version of FastAPI, what happens if you use the latest release instead? |
|
@teymour-aldridge haven't tried with latest version ? but i like to give it a shot, if there were any memory fix done with the versions after 0.20.0 |
|
@teymour-aldridge updated to latest version, will report back, once I see the result, Thanks |
|
@teymour-aldridge I checked on our staging, after updating to latest versions [fast api and uvicorn], the memory issue still exists |
|
@teymour-aldridge can this one work, i saw this in uvicorn documentation: |
|
@prav2019 I don't know, it might do. How many requests are you handling |
|
@teymour-aldridge or trying to add gunicorn!!! |
|
im running into the same issue - memory usage slowly builds over time, runnign on gunicorn with 4 uvicorn workers |
|
+1, seeing the same issue Gunicorn + uvicorn worker class |
|
Reading through the uvicorn code.. adding the max-requests effectively just restarts the server as soon as you hit some arbitrary number of requests? I can't find any good documentation on what this number should be either 500? 1000? 10k? 100k? If anyone has any experience/advice here, I'm all ears |
|
@curtiscook The max-requests restarts the service completely, we need to configure workers to keep one running always, when we restart another, was able to solve memory issue, but got into one more, now sometimes I get multiple requests to workers with same data and each worker creates new entry into database. |
|
@prav2019 So what exactly solved your OOM issue, was it setting the max-requests? |
|
Hi, I actually have not solved my memory leak issue but it's small enough to not be a huge concern. I'm also seeing the memory leak in other async processes so it might be an issue with long running event loops in async python?
That's what I thought it might do. Not really a great solution then :( |
|
I would have thought it would be pretty tricky to have a memory leak in Python. Perhaps an underlying issue in the interpreter (C is very good for memory leaks :D) or a C extension module? Anyway this has peaked my interest, so I'll try and investigate a little to see what's causing the problem. Is this only happening in Docker containers, or can it be reproduced across a number of devices? I'm not experiencing this issue on my machine, having left a FastAPI process running for a few days (nothing unusual happened). It's generally a good move to use ephemeral (short-lived) processes to run applications and regularly recycle them in order to reduce the risk/impact a memory leak (which tend to build up over time) can have. |
|
I haven't tested outside of docker containers/ heroku docker containers |
|
Ah. I'll try and do some profiling. Unfortunately my time is pretty scarce these days with the number of different projects I'm working on but fingers crossed. |
|
+1
uvicorn main:app --host 0.0.0.0 --port 8101 --workers 4 client call the function below per minute, and server memory usage slowly builds over time. |
|
update: I add to |
|
#1624 (comment) . The max request fixed OOM, as it restarts server, but it opened to lot of concurrency issues. |
|
did replacing def with async def helped, because I tried long back it didn't!!!!! |
|
If you're in a hurry and need a quick and temporary solution for now.
This helped me. You can get 10 simultaneous request where each worker will be restarted when request is finished. Thus the memory will be released. |
This is great. I'm 90% sure that the issue you found is the one that I was experiencing. As one of the early posters on this issue, I haven't noticed this issue anymore--but there were a few things that have happened since... Namely:
It's possible that I still have a memory leak, but it's not as detrimental as 2 years ago. Re:
I don't think this is a viable solution since I believe this blocks the event loop and relies on multiprocessing, which skips out on one of the major benefits of the ASGI server (not getting [thread/process]bound with a single worker) I don't know if @tiangolo has any thoughts? I feel like we're finally closer to being able to close this issue |
|
I'm having a massive leak with tensorflow + inference with dockerized fastapi + uvicorn server. anyone met that? (I'm on a machine with 120GB RAM) |
|
While it didn't completely solve memory pile up over time, using import os
host = os.getenv("HOST", "0.0.0.0")
port = os.getenv("PORT", "8000")
# Gunicorn config variables
loglevel = os.getenv("LOGLEVEL", "error")
workers = int(os.getenv("WORKERS", "2"))
bind = f"{host}:{port}"
errorlog = "-"
logconfig = "/logging.conf" |
|
@Xcompanygames Consider Using ONNX instead of TF as it's usually faster and more reliable. I'm having a memory leak, but i think is because the inference data stays on memory / gets dupped at some point. Update: I wasn't closing correctly the onnx inference session. |
|
Can confirm I am experiencing the same issue. Using The Docker container starts at around 105.8 MiB of RAM usage when fresh. After running a Locust swarm (40 Users) all hitting an endpoint that returns data ranging from 200KB to 8MB - the RAM usage of the Docker container grows (and sometimes shrinks, but mostly grows) until I get an OOM exception. The endpoint retrieves data from the Neo4J database and closes the driver connection cleanly each time. I had some success making the function I'm curious why this topic isn't more popular; surely everyone would be experiencing this. Perhaps we all notice it due to our endpoints returning enough data for us to notice the increase in usage, whereas the general user would most times only return a few KB at a time. Additional details: |
|
I've also experienced this, I don't really understand what causes it. |
|
@tiangolo Is there any updates on this? Thank you. |
|
I'm also noticing that memory keeps on increasing when hitting my service with 20 virtual users as part of a performance test locally. Am using
|
|
What is the deal with the original issue of not returning EDIT: oh, okay. I see. The author included the bug report template without editing it. Sorry for the noise. |
|
Many folks are affected by this issue so definitely something is happening, but it could as well be that the problem is in the user code and not in fastapi. So I suggest that, to make things easier for the maintainers, if you're affected by this issue
Memory issues are tricky but without a good reproducer, it will be impossible for the maintainers to declare whether this is still a problem or not, and if it is, to fix it. |
|
If this can help with debugging, using Via python from fastapi import FastAPI, Request
import time
app = FastAPI()
@app.middleware("http")
async def middle_1(request: Request, call_next):
return await call_next(request)
@app.middleware("http")
async def middle_2(request: Request, call_next):
return await call_next(request)
@app.middleware("http")
async def middle_3(request: Request, call_next):
return await call_next(request)
@app.middleware("http")
async def middle_4(request: Request, call_next):
return await call_next(request)
@app.middleware("http")
async def middle_5(request: Request, call_next):
return await call_next(request)
@app.get("/")
def read_root():
return {"Hello": "World"} |
|
same issue |
|
I see the same thing across all my services using different FastApi/Uvicorn/Gunicorn and Sentry SDK versions.
And is receiving short-lived requests that trigger longer but relatively short background tasks (not like I made myself a Celery out of it). Another thing I can think of is my ignorance around the subject of database connections where we go something like engine = create_async_engine(settings.database_dsn)
session_factory = sessionmaker(engine, expire_on_commit=False, class_=AsyncSession)
Session = async_scoped_session(session_factory, scopefunc=current_task)
Base = declarative_base()
@asynccontextmanager
async def get_db():
session = Session()
try:
yield session
await session.commit()
except:
await session.rollback()
raise
finally:
await session.close()and then use Using Gunicorn with the Here's the memory usage (the spike on 10/21 is where I increased a replica count and it immediately hogged a lot of memory): to put it in traffic context (not that any correlation can be seen): I wanted to limit requests for gunicorn to refresh the workers but I'm getting Please tell me if I can help i.e. by providing more data. |
|
We solved a memory leak on uvicorn 0.18.3, and there were some things solved on FastAPI since the mentioned versions. Can someone confirm this issue is still happening with the latest uvicorn and FastAPI? |
|
Had an OOM issue whilst running Uvicorn 0.17.2 using tiangolo's pre-built image. The issue hasn't reappeared after switching to a self-built image based on Python 3.10. |
|
There are 44 participants on this issue, maybe more subscribed. Can we please focus, and not bother people with information that is not relevant? If someone can reply my last question, I can proceed from there. |
I'm still having this issue, the first part of that (from roughly 85% utilization to 100%) was on FastAPI 0.85.0 and Uvicorn 0.17.6/Gunicorn 20.1.0/Starlette 0.20.4. The first big drop (down to ~65% utilization) was when my autoscaler kicked in from 1 to 4, and the second big drop (down to ~20%) was when I turned off autoscaling and upgraded a single instance from 512MB to 2GB of memory. Alongside the instance upgrade, I also updated FastAPI to 0.87.0, Uvicorn to 0.20.0, and Starlette to 0.21.0, and it seems like my memory usage is still creeping up (from a base usage of ~380MB to currently 525MB and still growing), which would've hit 100% memory utilization again if I stayed at the previous instance size). The instance is hosted by Render, and it does health check requests. |
|
Are you able to share an MRE? |
|
I upgraded to: and the issue still persists (you can see a restart when an instance reached 600mb RAM usage): pyproject.toml |
|
Are you able to share a minimal, reproducible, example? |
|
Would this simple-api sample help? As far as I can tell
|
I don't think the arguments used there are enough to discard those. Thanks for the MRE. 👍 EDIT: I still cannot reproduce it: lorosanu/simple-api#1 (comment). |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
and others
First check
Example
Here's a self-contained, minimal, reproducible, example with my use case:
Description
/.{"Hello": "World"}.{"Hello": "Sara"}.Environment
To know the FastAPI version use:
python -c "import fastapi; print(fastapi.__version__)"To know the Python version use:
Additional context
Tracemalloc gave insight on the lines , that are top consumers of memory: (top one seems to be below line in uvicorn)
/usr/local/lib/python3.6/site-packages/uvicorn/main.py:305:
Line:
loop.run_until_complete(self.serve(sockets=sockets))
The text was updated successfully, but these errors were encountered: