Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Immich CLI crash: TypeError: fetch failed | duplicate key value violates unique constraint #9411

Open
joaopedrocg27 opened this issue May 12, 2024 · 0 comments
Open

Immich CLI crash: TypeError: fetch failed | duplicate key value violates unique constraint #9411

joaopedrocg27 opened this issue May 12, 2024 · 0 comments

Comments

@joaopedrocg27
Copy link

The bug

When running immich upload -a -r -c 4 . I get the error:

Checking files | ████████████████████████████████████████ | 100% | ETA: 0s | 19739/19739 assets
Found 2891 new files and 16848 duplicates
Uploading assets | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 0% | ETA: 1h30m | 5.5 MB/29.7 GB
node:internal/deps/undici/undici:12502
      Error.captureStackTrace(err, this);
            ^

TypeError: fetch failed
    at node:internal/deps/undici/undici:12502:13
    at async uploadFile (file:///opt/homebrew/lib/node_modules/@immich/cli/dist/index.js:17924:20)
    at async file:///opt/homebrew/lib/node_modules/@immich/cli/dist/index.js:17883:28
    at async Promise.all (index 0)
    at async uploadFiles (file:///opt/homebrew/lib/node_modules/@immich/cli/dist/index.js:17880:7)
    at async upload (file:///opt/homebrew/lib/node_modules/@immich/cli/dist/index.js:17811:21) {
  [cause]: Error: write EPIPE
      at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:95:16) {
    errno: -32,
    code: 'EPIPE',
    syscall: 'write'
  }
}

Node.js v22.1.0```

### The OS that Immich Server is running on

Docker/Kubernetes

### Version of Immich Server

v1.103.1

### Version of Immich Mobile App

CLI ver. is 2.2.0 

### Platform with the issue

- [X] Server
- [ ] Web
- [ ] Mobile

### Your docker-compose.yml content

```YAML
immich:
  ## This chart relies on the common library chart from bjw-s
  ## You can find it at https://github.com/bjw-s/helm-charts/tree/main/charts/library/common
  ## Refer there for more detail about the supported values

  # These entries are shared between all the Immich components

  env:
    REDIS_HOSTNAME: '{{ printf "%s-redis-master" .Release.Name }}'
    DB_HOSTNAME: "{{ .Release.Name }}-postgresql"
    DB_USERNAME: "{{ .Values.postgresql.global.postgresql.auth.username }}"
    DB_DATABASE_NAME: "{{ .Values.postgresql.global.postgresql.auth.database }}"
    # -- You should provide your own secret outside of this helm-chart and use `postgresql.global.postgresql.auth.existingSecret` to provide credentials to the postgresql instance
    DB_PASSWORD: "{{ .Values.postgresql.global.postgresql.auth.password }}"
    IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}'

  image:
    tag: v1.103.1

  immich:
    persistence:
      library:
        existingClaim: "immich-library"

  # Dependencies

  postgresql:
    enabled: true
    global:
      storageClass: "nfs-main"
      postgresql:
        auth:
          username: immich
          database: immich
          password: immich
    primary:
      storageClass: "nfs-main"
      resourcesPreset: "small"
      initdb:
        scripts:
          create-extensions.sql: |
            CREATE EXTENSION cube;
            CREATE EXTENSION earthdistance;
            CREATE EXTENSION vectors;

    backup:
      enabled: true
      storageClass: "nfs-main"


  redis:
    enabled: true
    global:
      storageClass: "nfs-main"
    master:
      resourcesPreset: "nano"

  # Immich components

  server:
    enabled: true
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
      limits:
        cpu: 2
        memory: 1Gi

  microservices:
    enabled: true
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
      limits:
        cpu: 3
        memory: 5Gi

  machine-learning:
    enabled: true
    persistence:
      cache:
        enabled: true
        size: 10Gi
        type: pvc
        accessMode: ReadWriteMany
        storageClass: nfs-disposable
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
      limits:
        cpu: 2
        memory: 5Gi

Your .env content

.

Reproduction steps

run: immich  upload -a -r -c 4 . with a large dataset

Relevant log output

On Postgres I got a lot of these:

│ 2024-05-12 18:23:30.691 UTC [450821] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:30.703 UTC [451311] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:30.703 UTC [451311] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \x5202ea43f51006679195a4327e06144724c35cf9) already e │
│ 2024-05-12 18:23:30.703 UTC [451311] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:30.705 UTC [451491] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:30.705 UTC [451491] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \xbee93d1f13a8a6e1a6788890eed801b4084e85c2) already e │
│ 2024-05-12 18:23:30.705 UTC [451491] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:31.283 UTC [450821] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:31.283 UTC [450821] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \xb3778f545c834bec6cc829f653fb5e25db97ed67) already e │
│ 2024-05-12 18:23:31.283 UTC [450821] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:31.288 UTC [451061] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:31.288 UTC [451061] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \xaa0557b9744963fed2dc582e30f040dece456a05) already e │
│ 2024-05-12 18:23:31.288 UTC [451061] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:31.299 UTC [451491] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:31.299 UTC [451491] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \x4fea543626797875cf85fe36f2a59bd3a56b5920) already e │
│ 2024-05-12 18:23:31.299 UTC [451491] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:31.311 UTC [451311] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:31.311 UTC [451311] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \x7991ce388978a0f56460e675b5f6741d80b4a9c0) already e │
│ 2024-05-12 18:23:31.311 UTC [451311] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:31.727 UTC [451311] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:31.727 UTC [451311] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \xad9a565c36d5a8dbc44119e41f02607564719672) already e │
│ 2024-05-12 18:23:31.727 UTC [451311] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:31.773 UTC [451491] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:31.773 UTC [451491] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \x2d1bc71bcbc1f971b44b139711b15a84308d7018) already e │
│ 2024-05-12 18:23:31.773 UTC [451491] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:31.775 UTC [451311] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:31.775 UTC [451311] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \xa04d954d0269f625fcfc2b70a12df4d336d466fb) already e │
│ 2024-05-12 18:23:31.775 UTC [451311] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│ 2024-05-12 18:23:31.983 UTC [451491] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_library_checksum"                                                                                          │
│ 2024-05-12 18:23:31.983 UTC [451491] DETAIL:  Key ("ownerId", "libraryId", checksum)=(9687a050-4be2-46ef-ac3d-e3784fe0e48f, 20d6295d-2ee1-4272-b377-aeea0f330477, \x21217d848e388e4a5825b0367d9fc747793d3144) already e │
│ 2024-05-12 18:23:31.983 UTC [451491] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "libraryId", "deviceId", "type", "originalPath", "previewPath", "thumbnailPath", "thumbhash", "encodedVideoPath │
│

Additional information

No response
@joaopedrocg27 joaopedrocg27 changed the title Immich CLI crash: TypeError: fetch failed Immich CLI crash: TypeError: fetch failed | duplicate key value violates unique constraint May 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@joaopedrocg27