Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(snapshots): Replace Stream.toList and the for each cycle to Stream.forEachOrdered #12576

Conversation

aivinog1
Copy link
Contributor

@aivinog1 aivinog1 commented Apr 27, 2023

Description

I replaced the collection with a list and the for each cycle to the Stream.forEactOrdered call. It should reduce memory footprint and possibly improve latency, especially on a large number of snapshot files.

Related issues

closes #12575

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

  • The changes are backwards compatibility with previous versions
  • If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

  • There are unit/integration tests that verify all acceptance criterias of the issue
  • New tests are written to ensure backwards compatibility with further versions
  • The behavior is tested manually
  • The change has been verified by a QA run
  • The impact of the changes is verified by a benchmark

Documentation:

  • The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
  • If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Other teams:
If the change impacts another team an issue has been created for this team, explaining what they need to do to support this change.

Please refer to our review guidelines.

@aivinog1
Copy link
Contributor Author

Hello, reviewer!
First of all, this should be benchmarked with a large number of snapshot files, so that we could see the difference in the memory and snapshot creation latency.
Then, it should be nice to backport this in the 8.0, 8.1, and 8.2 as well.

@deepthidevaki deepthidevaki self-requested a review May 3, 2023 08:24
Copy link
Contributor

@deepthidevaki deepthidevaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Thanks @aivinog1 for your contribution.

Since this is reasonable change irrespective of performance improvement, I'm merging this without running a benchmark. You would see a difference only with large number of snapshot files anyway.

bors r+

zeebe-bors-camunda bot added a commit that referenced this pull request May 4, 2023
12576: refactor(snapshots): Replace `Stream.toList` and the for each cycle to `Stream.forEachOrdered` r=deepthidevaki a=aivinog1

## Description

<!-- Please explain the changes you made here. -->
I replaced the collection with a list and the for each cycle to the `Stream.forEactOrdered` call. It should reduce memory footprint and possibly improve latency, especially on a large number of snapshot files.

## Related issues

<!-- Which issues are closed by this PR or are related -->

closes #12575 



Co-authored-by: Alexey Vinogradov <vinogradov.a.i.93@gmail.com>
@deepthidevaki deepthidevaki added backport stable/8.0 backport stable/8.2 Backport a pull request to 8.2.x labels May 4, 2023
@zeebe-bors-camunda
Copy link
Contributor

Build failed:

@deepthidevaki
Copy link
Contributor

Smoke test failed.

bors retry

@zeebe-bors-camunda
Copy link
Contributor

Build succeeded:

@zeebe-bors-camunda zeebe-bors-camunda bot merged commit 2fdfca9 into camunda:main May 4, 2023
36 checks passed
@aivinog1 aivinog1 deleted the feature/12575_improve_snapshots_traversion branch May 4, 2023 10:24
@aivinog1
Copy link
Contributor Author

aivinog1 commented May 4, 2023

Thanks @deepthidevaki 🎉
Could you approve the backport workflow, please? https://github.com/camunda/zeebe/actions/runs/4880268131
It will not create backport PRs without approval :(

@deepthidevaki
Copy link
Contributor

@aivinog1 backport workflow is running. Please assign it to me when the backport PRs are created.

@aivinog1 aivinog1 restored the feature/12575_improve_snapshots_traversion branch May 4, 2023 10:51
@aivinog1
Copy link
Contributor Author

aivinog1 commented May 4, 2023

@aivinog1 backport workflow is running. Please assign it to me when the backport PRs are created.

@deepthidevaki It failed (I think because I drop my branch, but I just restored it now), could you restart the workflow, please? 🙂 Or check out what went wrong.

@deepthidevaki
Copy link
Contributor

Let's try again.

/backport

@backport-action
Copy link
Collaborator

Backport failed for stable/8.0, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git fetch origin stable/8.0
git worktree add -d .worktree/backport-12576-to-stable/8.0 origin/stable/8.0
cd .worktree/backport-12576-to-stable/8.0
git checkout -b backport-12576-to-stable/8.0
ancref=$(git merge-base 02d1df8ac1b6f5484f1e9d4c1f19c8a5712176b1 499036719908a9487853a727c54e04c637119498)
git cherry-pick -x $ancref..499036719908a9487853a727c54e04c637119498

@backport-action
Copy link
Collaborator

@backport-action
Copy link
Collaborator

zeebe-bors-camunda bot added a commit that referenced this pull request May 4, 2023
12666: [Backport stable/8.2] refactor(snapshots): Replace `Stream.toList` and the for each cycle to `Stream.forEachOrdered` r=deepthidevaki a=backport-action

# Description
Backport of #12576 to `stable/8.2`.

relates to #12575

Co-authored-by: Alexey Vinogradov <vinogradov.a.i.93@gmail.com>
zeebe-bors-camunda bot added a commit that referenced this pull request May 4, 2023
12662: [Backport stable/8.1] fix: do not retake backup if it already exists r=deepthidevaki a=deepthidevaki

Backport of #12626 to stable/8.1.

relates to #12623

12665: [Backport stable/8.1] refactor(snapshots): Replace `Stream.toList` and the for each cycle to `Stream.forEachOrdered` r=deepthidevaki a=backport-action

# Description
Backport of #12576 to `stable/8.1`.

relates to #12575

Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>
Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@users.noreply.github.com>
Co-authored-by: Alexey Vinogradov <vinogradov.a.i.93@gmail.com>
@oleschoenburg oleschoenburg added the version:8.2.5 Marks an issue as being completely or in parts released in 8.2.5 label May 16, 2023
@oleschoenburg oleschoenburg added the version:8.1.13 Marks an issue as being completely or in parts released in 8.1.13 label Jun 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport stable/8.2 Backport a pull request to 8.2.x version:8.1.13 Marks an issue as being completely or in parts released in 8.1.13 version:8.2.5 Marks an issue as being completely or in parts released in 8.2.5
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve the traversing of snapshot files
4 participants