Asset is offline - but it is not

[psla]

The bug

Immich claims that it can't access the file (this seems to be the case for all the files that I tested from the external library)

however, when I log in to immich's shell, the file is found and the permissions look accessible

root@cf8d6f9c93d1:/usr/src/app# ls -al /mnt/media/Fotografie/2022/2022-10-29_to_30_Yakima/Pixel5/PXL_20221029_194225561.jpg

-r-xr--r-- 1 nobody nogroup 3341292 Oct 29  2022 /mnt/media/Fotografie/2022/2022-10-29_to_30_Yakima/Pixel5/PXL_20221029_194225561.jpg

The OS that Immich Server is running on

Linux cf8d6f9c93d1 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

Version of Immich Server

v1.94.1

Version of Immich Mobile App

n/a

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.3"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/nas/public/wspolne/Fotografie:/mnt/media/Fotografie:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/nas/public/wspolne/Fotografie:/mnt/media/Fotografie:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
  backup:
    container_name: immich_db_dumper
    image: prodrigestivill/postgres-backup-local
    env_file:
      - .env
    environment:
      POSTGRES_HOST: database
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      SCHEDULE: "@daily"
      BACKUP_NUM_KEEP: 2
      BACKUP_DIR: /db_dumps
    volumes:
      - /mnt/newhome/unprivileged-docker/immich/db_dumps:/db_dumps
    depends_on:
      - database

volumes:
  pgdata:
  model-cache:

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=REDACTED
DB_DATABASE_NAME=immich



REDIS_HOSTNAME=immich_redis






UPLOAD_LOCATION=/mnt/nas/srv/immich/pictures


TYPESENSE_API_KEY=GiY1Up1U




PUBLIC_LOGIN_PAGE_MESSAGE=


IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003


IMMICH_API_URL_EXTERNAL=https://REDACTED/

Reproduction steps

1. go to immich web
2. find a file from the library
3. observe the red text
...

Additional information

I forced-rescanned the library, it did not help

Please let me know what logs I can provide to help diagnose. I looked at docker logs but nothing stood out.

[alextran1502]

cc @etnoy

[etnoy]

Inside the docker container, can you read the file? Like try to execute the file command:

file /mnt/media/Fotografie/2022/2022-10-29_to_30_Yakima/Pixel5/PXL_20221029_194225561.jpg

[psla]

root@cf8d6f9c93d1:/usr/src/app# head -c1 /mnt/media/Fotografie/2022/2022-10-29_to_30_Yakima/Pixel5/PXL_20221029_194225561.jpg
�

(file command does not exist inside of the container, but the file is read successfully)

[psla]

root@cf8d6f9c93d1:/usr/src/app# head -c1 /mnt/media/Fotografie/2022/2022-10-29_to_30_Yakima/Pixel5/PXL_20221029_194225561.jpg
�

reading works fine

[etnoy]

Can you verify that both the server container and microservices containers can read the files?

[ldivl]

The forced rescan did resolve issue. Really don't understand why it was required, but it did work.

[psla]

For me, forced rescan did not resolve the issue.

unprivileged-docker@usa:~/immich$ docker exec -it immich_microservices bash
root@088dd6682fe0:/usr/src/app# wc -c /mnt/media/Fotografie/2015/2015-07_Wyjazd/2015-07_Tata/WP_20150718_11_03_31_Pro.jpg

3125231 /mnt/media/Fotografie/2015/2015-07_Wyjazd/2015-07_Tata/WP_20150718_11_03_31_Pro.jpg

unprivileged-docker@usa:~$ docker exec -it immich_server bash
root@6065d52b710d:/usr/src/app# wc -c /mnt/media/Fotografie/2015/2015-07_Wyjazd/2015-07_Tata/WP_20150718_11_03_31_Pro.jpg

3125231 /mnt/media/Fotografie/2015/2015-07_Wyjazd/2015-07_Tata/WP_20150718_11_03_31_Pro.jpg

Trying the force rescan again. Also, doing "scan new library files" is not really finding the new files

Is there a way to enable some debug/verbose logging?

[psla]

So the full refresh logs this line:

[Nest] 7  - 05/12/2024, 8:40:35 PM     LOG [LibraryService] Refreshing library: 27af95c1-57fc-4885-9b55-f78a9bc36276

But it doesn't seem like it's doing much. when I run lsof on my file system, it doesn't seem like immich is doing anything.

When I tried to run a storage migration, I got

[Nest] 7  - 05/12/2024, 8:01:30 PM    WARN [MediaService] Unable to complete move. Error renaming file with code ENOENT and message: ENOENT: no such file or directory, rename 'upload/thumbs/d38cadab-260f-4e27-8c8a-a11a7a627894/fe/46/fe468c9c-c78b-4787-8c0c-86c8556a0597.jpeg' -> 'upload/thumbs/d38cadab-260f-4e27-8c8a-a11a7a627894/fe/46/fe468c9c-c78b-4787-8c0c-86c8556a0597-thumbnail.jpeg'

which I assume is unrelated, hopefully.

There was never a log that refresh completed, but there are also no outstanding tasks in the jobs queue.

[Nest] 7  - 05/12/2024, 6:51:27 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.103.1] [PRODUCTION]
[Nest] 7  - 05/12/2024, 6:58:03 PM     LOG [LibraryService] Refreshing library: 27af95c1-57fc-4885-9b55-f78a9bc36276
[Nest] 7  - 05/12/2024, 7:04:24 PM     LOG [LibraryService] Refreshing library: 27af95c1-57fc-4885-9b55-f78a9bc36276
[Nest] 7  - 05/12/2024, 8:01:30 PM    WARN [MediaService] Unable to complete move. Error renaming file with code ENOENT and message: ENOENT: no such file or directory, rename 'upload/thumbs/d38cadab-260f-4e27-8c8a-a11a7a627894/fe/46/fe468c9c-c78b-4787-8c0c-86c8556a0597.jpeg' -> 'upload/thumbs/d38cadab-260f-4e27-8c8a-a11a7a627894/fe/46/fe468c9c-c78b-4787-8c0c-86c8556a0597-thumbnail.jpeg'
[Nest] 7  - 05/12/2024, 8:40:35 PM     LOG [LibraryService] Refreshing library: 27af95c1-57fc-4885-9b55-f78a9bc36276

I am on v1.103.1

[psla]

One more observation, when I tapped "refresh metadata" and "refresh thumbnail" for such file, it gets refreshed correctly:

[Nest] 7  - 05/12/2024, 9:01:56 PM     LOG [MediaService] Successfully generated JPEG image preview for asset 45070b93-3cf6-43ed-b802-384262b2bda6
[Nest] 7  - 05/12/2024, 9:01:56 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 45070b93-3cf6-43ed-b802-384262b2bda6

[matticas]

I'm in the same boat, "Assets offline" when I have confirmed all the containers can see the external library. It's been this way for the last few releases,

Forced Rescan doesn't appear to actually start. On the Jobs page, the Library Tasks clicks up to '1' for a very brief moment then goes back to '0'.

[rvoosterhout]

Same issue here. I have a lot of duplicates in my library. I believe they were created by running the Migration job, but I'm not sure about that one. Every duplicate image has the 'Asset offline' message. If there is anything I can do to test and help, I'd be happy to.

[matticas]

Same issue here. I have a lot of duplicates in my library. I believe they were created by running the Migration job, but I'm not sure about that one. Every duplicate image has the 'Asset offline' message. If there is anything I can do to test and help, I'd be happy to.

We may be in the same boat, but I do not have duplicates;

#9574