Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Dynamic Schema Update for Elastic Search #18486

Open
Tracked by #18484
nathansandi opened this issue May 14, 2024 · 0 comments · May be fixed by #18618
Open
Tracked by #18484

Implement Dynamic Schema Update for Elastic Search #18486

nathansandi opened this issue May 14, 2024 · 0 comments · May be fixed by #18618
Assignees
@nathansandi @esraagamal6
Labels
component/backend Related to a backend component component/tasklist Related to the Tasklist component/team

Comments

@nathansandi
Copy link
Contributor

nathansandi commented May 14, 2024 •
edited

Description

We wanna have dynamic schema update in order to reduce the downtime, and replace the data migration strategy.

Solution

  1. Verify difference over the index
  2. In the case of index difference, the indices should be updated

The update should be applied over the Schema Startup on the application

Acceptance Criteria

  • Schema is updated after a change
  • Start/Update/Complete a Task should work fine after a schema update

PS:

  • Remove the migration code is out of scope of this ticket (But a condition should be added). We will keep for while the migration code
@nathansandi nathansandi self-assigned this May 14, 2024
@nathansandi nathansandi added the component/backend Related to a backend component label May 14, 2024
@nathansandi nathansandi mentioned this issue May 14, 2024
Open
@nathansandi nathansandi added the component/tasklist Related to the Tasklist component/team label May 14, 2024
This was referenced May 21, 2024
Merged
Merged
Open
nathansandi added a commit that referenced this issue May 23, 2024
@nathansandi
test: Dynamic schema update Integration Tests (Elastic Search) (#18623)
c4f7a9e 
## Description

Integration test classes for Dynamic Schema Updates.

Implementation Tickets:
Part 1: #18618 
Part 2: #18622

## Related issues

closes #18486
nathansandi added a commit that referenced this issue May 23, 2024
@nathansandi
feat: Implement dynamic schema update using index mapping difference (#…
e23bfc4 
…18622)

## Description

- Retrieve the Index/Template Mappings over RetryElasticSearchclient 
- Placeholders for the OpenSearch implementation 
- Update the Schema Startup to consider the Update Schema setup 
- Updated the (Abstract) Index descriptors to return
getSchemaClasspathFilename() and getAllVersionsIndexNameRegexPattern()
- Update ElastciSearch Schema Manager to Return the Expected Index
Fields from the mapping, considering the differences
- Adjust Schema Manager for create Index and Template making the public
as well in order to use the methods on test
- Extract ElasticSearchReadJson Utill to a different method (Refactor)


Its the second part of the implementation:
Part 1: #18618 

## Related issues

closes #18486
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nathansandi nathansandi

@esraagamal6 esraagamal6

Labels
component/backend Related to a backend component component/tasklist Related to the Tasklist component/team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: dynamic schema update feature PR camunda/camunda
2 participants
@nathansandi @esraagamal6