-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Possible behavior mismatch: Immich resets vector length in smart_search database when model is changed via UI, but does not perform this check if model is specified via config file #9398
Comments
That sounds correct, and would explain some issues we've seen. @mertalev thoughts? |
It does (or should I guess) check if it should update the smart search table at startup. Specifically, it checks if the dimensions of the current model match the dimensions in the table and updates the table if not. It's possible there's a bug somewhere in that logic, though. |
Apologies if I'm completely off on the wrong path, I don't know typescript at all or Javascript basically at all, but in the codebase I'm seeing the following control flow: (inside the
At this point, the following seems to occur:
Taking a brief break from bullet points to refocus: at this point we're at Line 36 of the database.service.ts file. At this point, the following happens:
I am probably missing some sort of |
The check happens in the search repository's init, referenced in the smart info service. |
Oh, I guess the init in the smart info service isn't getting called anywhere haha. |
Ah, so it’s just not calling that unless you change the model while running? |
Yup, so it only updates on a config change event in practice. Making it run at startup should fix it. |
The bug
When updating the CLIP model, Immich Server is expected to update the database table "smart_search" to set the data type of the "embedding" column to a vector width which matches the CLIP model selected.
I believe that currently this behavior is only being performed when that change is made IN THE WEB UI, and is not performed at system startup or during the config-file parsing, even if the CLIP model was changed.
It is possible that this behavioral issue is restricted to environments where there is no persistence (e.g. deployed as a pod in Kubernetes without an underlying persistent volume), and does not appear if the immich-server has a stable data source beneath it. I believe that performing this check on startup when using a config file would still be a more robust behavior.
The OS that Immich Server is running on
k8s on Ubuntu 22.04
Version of Immich Server
v1.103.1
Version of Immich Mobile App
N/A
Platform with the issue
Your docker-compose.yml content
Deployed from k8s yaml files
Your .env content
Reproduction steps
Relevant log output
Additional information
Background
I stood up a new immich instance using Kubernetes.
In doing so, I initially brought the server pod up without a config file specified.
I changed several config variables and exported the JSON. While doing this, I did NOT change the "Clip Model" under Machine Learning Settings.
I added the exported JSON to a K8s ConfigMap and updated the Pod to mount that ConfigMap as a directory.
I updated the pod environment (for Server and Microservices) to include the "IMMICH_CONFIG_FILE" variable pointing to the mounted config file, and restarted the pods so they would use that config file. At this point the server came up and behaved normally, and exhibited no errors in the logs of any component.
I verified that the Server was indeed using the config file, as the "Settings" page of Administration stated that and locked any changes from being made.
I then edited the config file definition to change the Clip model to "ViT-g-14__laion2b-s12b-b42k".
I applied this change to the k8s configmap and restarted the server instance. it came up correctly, and showed the new model name in the "Settings" of the web administration tool. No errors or unusual messages were present in the log at this time.
Error detected
I then ran the "smart search" job for All assets (since I had changed the clip model). The "waiting" count did not go down, and error logs from the microservices pod indicated "the given vector is invalid for input" (see logs section).
Research
I found this discussion indicating a similar issue: #7425
The error seems to be directly related to a mismatch between the vector size of the "embedding" column in the smart_search table of the database vs. the vector size the CLIP model requires.
Resolving the issue requires making a change to the data type of the column (setting the vector size correctly), and rebuilding some indexes on the table.
A commenter in that discussion stated that the immich server should be performing these database changes when it detects that the size of the model has changed, and included two server log lines indicating such:
These lines never appeared in my server logs.
I then disabled the IMMICH_CONFIG_FILE environment variable and restarted the immich server pod. This reset the settings to defaults, including the CLIP model.
I then changed the model IN THE ADMIN WEB UI to the desired one and clicked save. The server immediately performed the expected database update.
I re-enabled the IMMICH_CONFIG_FILE environment variable and restarted the pod again. I was then able to perform a smart-search reindex using the desired CLIP model. successfully.
It's possible that this behavior/issue is limited to cases where a config file is being mounted into a pod via a Kubernetes ConfigMap, or cases where there is no persistence for immich-server except the config file.
The text was updated successfully, but these errors were encountered: