Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model update resets document auto-generated embeddings #76

Open
ingria opened this issue Aug 17, 2023 · 1 comment
Open

Model update resets document auto-generated embeddings #76

ingria opened this issue Aug 17, 2023 · 1 comment

Comments

@ingria
Copy link

ingria commented Aug 17, 2023

Description

In Typesense version 0.25 it's possible to define an embedding field. But since this field is auto-generated and is not stored locally, any update to the model will cause the embeddings field to reset.

Steps to reproduce

Schema:

    {
      "name": "embedding",
      "type": "float[]",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "embed": {
        "from": [
          "some_field"
        ],
        "model_config": {
          "model_name": "ts/paraphrase-multilingual-mpnet-base-v2"
        }
      },
      "num_dim": 768
    }

After I import the model, Typesense takes some time to generate embeddings. After that process, all documents will have the embedding field with array of 768 floats.

Then, If I call searchable() method on the model, the embedding field becomes empty.

Expected Behavior

embedding field should ether be updated if embed.from fields are changed, or be left unchanged.

Actual Behavior

the embedding field becomes empty

Metadata

Typesense Version: 0.25.0

OS: Ubuntu 20.04

@ingria
Copy link
Author

ingria commented Aug 17, 2023

Also, subsequent calls of artisan scout:import command deletes the embedding field on all of the models in Typesense collection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant