Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: Add memory size for binlog #33025

Merged
merged 8 commits into from
May 15, 2024

Conversation

xiaocai2333
Copy link
Contributor

issue: #33005

  1. add MemorySize field for insert binlog.
  2. LogSize means the file size in the storage object.
  3. MemorySize means the size of the data in the memory.

@sre-ci-robot sre-ci-robot added the size/M Denotes a PR that changes 30-99 lines. label May 13, 2024
Copy link
Contributor

mergify bot commented May 13, 2024

@xiaocai2333

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

  1. Title Format: The PR title must begin with one of these prefixes:
  • feat: for introducing a new feature.
  • fix: for bug fixes.
  • enhance: for improvements to existing functionality.
  • test: for add tests to existing functionality.
  • doc: for modifying documentation.
  • auto: for the pull request from bot.
  1. Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly 

Please review and update your PR to comply with these guidelines.

@xiaocai2333 xiaocai2333 changed the title Add memory size for insert binlog enhance: Add memory size for insert binlog May 13, 2024
@mergify mergify bot added kind/enhancement Issues or changes related to enhancement and removed do-not-merge/invalid-pr-format labels May 13, 2024
Copy link
Contributor

mergify bot commented May 13, 2024

@xiaocai2333 ut workflow job failed, comment rerun ut can trigger the job again.

Copy link

codecov bot commented May 13, 2024

Codecov Report

Attention: Patch coverage is 90.76923% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 82.08%. Comparing base (1d48d0a) to head (a44b154).
Report is 8 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #33025      +/-   ##
==========================================
- Coverage   82.14%   82.08%   -0.06%     
==========================================
  Files        1005     1005              
  Lines      128103   128155      +52     
==========================================
- Hits       105224   105201      -23     
- Misses      18911    18964      +53     
- Partials     3968     3990      +22     
Files Coverage Δ
internal/datacoord/compaction_trigger.go 83.77% <100.00%> (ø)
internal/datacoord/compaction_view.go 88.31% <100.00%> (+11.03%) ⬆️
internal/datacoord/segment_info.go 87.30% <100.00%> (ø)
internal/datacoord/util.go 99.13% <100.00%> (ø)
internal/datanode/binlog_io.go 91.86% <100.00%> (+0.06%) ⬆️
internal/datanode/syncmgr/task.go 100.00% <100.00%> (ø)
internal/metastore/kv/datacoord/kv_catalog.go 81.72% <100.00%> (+0.07%) ⬆️
internal/querynodev2/segments/index_attr_cache.go 100.00% <100.00%> (ø)
internal/querynodev2/segments/segment_loader.go 73.62% <100.00%> (+0.12%) ⬆️
internal/querynodev2/services.go 91.04% <100.00%> (-0.34%) ⬇️
... and 5 more

... and 32 files with indirect coverage changes

@mergify mergify bot added the ci-passed label May 13, 2024
@czs007 czs007 added this to the 2.4.2 milestone May 14, 2024
internal/proto/data_coord.proto Show resolved Hide resolved
internal/datacoord/util.go Outdated Show resolved Hide resolved
internal/querynodev2/segments/segment_loader.go Outdated Show resolved Hide resolved
internal/storage/data_codec.go Outdated Show resolved Hide resolved
@sre-ci-robot sre-ci-robot added size/L Denotes a PR that changes 100-499 lines. and removed size/M Denotes a PR that changes 30-99 lines. labels May 14, 2024
@mergify mergify bot removed the ci-passed label May 14, 2024
Copy link
Contributor

mergify bot commented May 14, 2024

@xiaocai2333 ut workflow job failed, comment rerun ut can trigger the job again.

Copy link
Contributor

mergify bot commented May 14, 2024

@xiaocai2333 ut workflow job failed, comment rerun ut can trigger the job again.

@chyezh
Copy link
Contributor

chyezh commented May 14, 2024

/lgtm

internal/datacoord/compaction_trigger.go Show resolved Hide resolved
internal/proto/data_coord.proto Outdated Show resolved Hide resolved
Copy link
Contributor

@XuanYang-cn XuanYang-cn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. l0_compactor.go
  • GetLogSize() -> GetMemSize()
  • Set MemSize in L:345
  1. data_codec.go
  • L1021: Set blobs MemSize by deleteData.Size()

@xiaocai2333 xiaocai2333 changed the title enhance: Add memory size for insert binlog enhance: Add memory size for binlog May 14, 2024
@XuanYang-cn XuanYang-cn self-requested a review May 14, 2024 06:57
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Copy link
Contributor

mergify bot commented May 14, 2024

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Copy link
Contributor

mergify bot commented May 14, 2024

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333
Copy link
Contributor Author

/run-cpu-e2e

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
@xiaofan-luan
Copy link
Contributor

/lgtm
/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xiaocai2333, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
@sre-ci-robot sre-ci-robot removed the lgtm label May 14, 2024
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Copy link
Contributor

mergify bot commented May 14, 2024

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333
Copy link
Contributor Author

/run-cpu-e2e

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
@czs007
Copy link
Contributor

czs007 commented May 15, 2024

/lgtm

@mergify mergify bot added the ci-passed label May 15, 2024
@sre-ci-robot sre-ci-robot merged commit 6ea7633 into milvus-io:master May 15, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement lgtm size/L Denotes a PR that changes 100-499 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants