[Tracking-issue] Settings diff-indexing

[ManyTheFish]

Related product team resources: PRD (internal only)

Motivation

The indexing process is split into several parts, extracting different data from the documents depending on the settings.
Each extraction can be precisely linked to one or several settings when running indexing, but Meilisearch v1.7 and older versions reindex everything like it was fresh indexing.
So, even if the original documents have not been changed, Meilisearch will delete them to rewrite the same data with different internal IDs.
Changing settings should only reindex the data related to the changes.
This issue lists all the issues related to the diff-indexing subject.

TODO

P1:

• Avoid a full index deletion in Transform when changing settings #4478
• Create SettingsDiff structure and run extractions based on it #4480

P2:

• Swapping searchableAttributes shouldn't trigger a reindexing #4484
• Reduce Transform's disk usage when changing the settings #4485
• Adding a new searchableAttribute should not re-index all the attributes #4492

P3:

• Process word-pair-proximity database differently depending on the Settings #4482

• Modifying an embedder should not always recompute all the vectors #4615

• Remove facet database when an attribute is removed from the facet list #4490

• Update the spec

Impacted teams

• @meilisearch/docs-team: no API changes, but solving this issue will no longer trigger a full reindexing when changing the settings, we may remove any mention of it.

[dureuill]

Hello, some information on the specific case of embedders.

1. Changes to 1 embedder should not trigger the reindexing of all embedders. Only the modified embedder should be reindexed
   • In the case where one embedder is deleted, the code currently benefits from the "reindex everything" behavior in that the associated index of embedders after the deleted embedder is modified to not leave a "hole" in the list of embedders. A free list is required to support incremental deleting of embedder.
2. Currently, changes to searchable, displayable, sortable, filterable, etc. do not need to trigger a reindexing. In the future, the properties of a field might be accessible to the document template, madking reindexing mandatory in this case.
3. A change to the apiKey (for OpenAI models) should not trigger a reindexing operation of the modified embedder.
4. When modifying the documentTemplate, a reindexing is necessary, but would be more minimal if comparing the rendered versions of both the old and the new template, and only regenerating embeddings for documents where the rendered version actually changed.
5. Future extensions:
   1. distributionShift: should not trigger a reindexing operation.
   2. distance: needs reindexing