[Bug] immich microservice memory leak kills host #9414
Comments
Dunky-Z
changed the title
|
How much RAM does the server have, and how much is used just after starting thumbnail generation? An increase in memory usage for some period of time during thumbnail generation is normal because of memory fragmentation, but it should plateau after a certain point. If you have any RAW images, they will amplify this effect since they require much more memory. |
|
Thank you for your response. The server has a total of 16GB of RAM, and its typical daily memory usage hovers around 6GB. After initiating thumbnail generation, it doesn't immediately consume all memory but gradually fills it up until the program crashes. Your mention of RAW images did remind me that the majority of my gallery consists of RAW files. Is there a way to prevent excessive memory usage in this scenario? I have tried limiting the microservice using Cgroups, but when the limit is exceeded, the microservice restarts. Since I bind the Cgroup settings to the PID, once the microservice restarts, PID changed, the previous Cgroup configurations become ineffective. |
|
Given the server's average performance, I have configured all my JOBS to use a single thread, and only one JOB is executed at a time. In theory, processing a single RAW file shouldn't require such an excessive amount of memory. |
|
Hmm, that much of an increase is unexpected. It could be related to the issue behind #6542, or possibly #4391. Are there any errors in the microservices logs before the OOM error? As far as limiting memory usage in the meantime, you can set a limit through Docker, which will force the container to be restarted after reaching a certain usage. |
|
Below are some logs from the microservices; perhaps they might provide some insight: [Nest] 7 - 05/13/2024, 11:34:38 AM LOG [EventRepository] WebSocket server initialized.
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: The input file contains an unsupported image format
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Error: The input file contains an unsupported image format
at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
at MediaRepository.resize (/usr/src/app/dist/repositories/media.repository.js:76:14)
at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:156:48)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:134:29)
at async /usr/src/app/dist/services/job.service.js:149:36
at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Affected Job ID:
{
"id": "5802ddbb-4e01-47fb-b21b-73bc0361f9e8"
}
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
Built with gcc 12 (Debian 12.2.0-14)
Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
Built with gcc 12 (Debian 12.2.0-14)
Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input
at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
at ChildProcess.emit (node:events:518:28)
at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Affected Job ID:
{
"id": "ee18e5e8-5430-4fcb-aa10-ef4170e2b30e"
}Limiting the container's resource usage has indeed helped me; it prevents my server from crashing. However, upon examining the microservices' logs, I noticed frequent restarts, indicating that the microservices are indeed exceeding the memory limit. Even after setting a 4GB memory limit, it's still insufficient. Here's the relevant part of my Docker Compose configuration: immich-microservices:
container_name: immich_microservices
image: altran1502/immich-server:${IMMICH_VERSION:-release}
command: ['start.sh', 'microservices']
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- "/root/sharedfolder/syncthing/Photo_Album:/mnt/media/Photo_Album:ro"
- /etc/localtime:/etc/localtime:ro
devices:
- /dev/dri:/dev/dri
env_file:
- .env
depends_on:
- redis
- database
restart: always
deploy:
resources:
limits:
cpus: '2'
memory: 4G[Nest] 7 - 05/13/2024, 11:35:19 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:33 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:45 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:35:58 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:36:11 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:36:53 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:06 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:19 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:34 AM LOG [EventRepository] Initialized websocket server
[Nest] 7 - 05/13/2024, 11:37:47 AM LOG [EventRepository] Initialized websocket serverI'm still hoping to identify and resolve the root cause entirely. If more information is needed, please let me know. On a side note, how can I persistently save Immich logs? The log path isn't mentioned in the user documentation, so I haven't mapped any volumes, resulting in log loss upon container restarts. Thank you very much！ |
|
I have the same problem (i5 8th gen server, running OMV 6) - am also Looking for a easy way to save immich logs - |
|
|
|
Thanks for the detailed info!
|
|
How many of those assets are RAW, if any? 50% of the images are RAW around 50GB each. I just restarted the container. That smooth section is unusual, and it will go back to the sawtooth. |
|
Below is an example of the errors I was receiving on the NAS when 4GB was installed and getting constant restarts of the microservices container. |
|
I made a test image for microservices with a possible fix: |
|
Not good news I am afraid. The immich_microservices fails. immich_microservices 2024/05/23 01:09:52 stderr } 2024/05/23 01:09:52 stderr routine: 'parserOpenTable' 2024/05/23 01:09:52 stderr line: '1381', 2024/05/23 01:09:52 stderr line: '1381', |
|
Hmm, that error is about connecting to Postgres, not related to thumbnail generation. |
|
Yeah... Something may have gone wrong with the deployment. Let me do some more testing, and maybe get some sleep :-). |
|
So sorry! I based that branch off of the latest release, but it turns out that main gets merged into it anyway when the image is built. The error is probably because of that. You can either wait for the next release to get things back up or restore from a backup. (It's also possible to mess with it more to get it back up, but I think these options are safer.) |
|
Thanks for letting me know and I can confirm I am getting the same result. |
The bug
Today, I encountered the same issue #5283 . After bulk importing around 10,000 photos(By setting the external library), I suddenly couldn't remotely connect to my NAS host. Initially, I suspected that the ML service was causing high memory usage during facial recognition, so I forcibly restarted the NAS and reconfigured concurrency settings, setting all jobs to run on a single thread. I didn't start all the jobs at once; instead, I sequentially ran FACE DETECTION and then GENERATE THUMBNAILS. There were no abnormalities during the FACE DETECTION process, indicating that the issue wasn't caused by the ML operations. After completing all FACE DETECTIONS, I initiated the GENERATE THUMBNAILS task, which ran slowly due to the large image sizes. Without continuous monitoring, upon returning to check on the NAS, I found it was unreachable remotely, with the system log displaying the following error messages:
This suggests that there might be a memory leak issue within the immich_microser service. I would greatly appreciate any advice on how to address this problem.
The OS that Immich Server is running on
Debian(OMV)
Version of Immich Server
OpenMediaVault 5
Version of Immich Mobile App
v1.103.1
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Relevant log output
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: