[VL] Refine memory changes logic when growCapacity #5690
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
4 similar comments
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
/Benchmark Velox |
| int64_t reserve(int64_t diff) { | ||
| std::lock_guard<std::mutex> lock(mutex_); | ||
| bytesReserved_ += diff; | ||
| int64_t newBlockCount; | ||
| if (bytesReserved_ == 0) { | ||
| newBlockCount = 0; | ||
| } else { | ||
| // ceil to get the required block number | ||
| newBlockCount = (bytesReserved_ - 1) / blockSize_ + 1; | ||
| } | ||
| int64_t bytesGranted = (newBlockCount - blocksReserved_) * blockSize_; | ||
| blocksReserved_ = newBlockCount; | ||
| maxBytesReserved_ = std::max(maxBytesReserved_, bytesReserved_); | ||
| return bytesGranted; | ||
| } |
There was a problem hiding this comment.
Does VeloxMemoryManager::arrowPool_ still use this code? Can we move it into another independent gluten::AllocationListener implementation which can wrap another listener? Like the way BacktraceAllocationListener adopted.
There was a problem hiding this comment.
I move such logic to ListenableMemoryAllocator.
|
Run Gluten Clickhouse CI |
1 similar comment
|
Run Gluten Clickhouse CI |
Yohahaha
force-pushed
the
optimize-arbitrator
branch
from
4e44a94
to
09db8a9
Compare
|
Run Gluten Clickhouse CI |
Yohahaha
force-pushed
the
optimize-arbitrator
branch
from
09db8a9
to
922e291
Compare
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
/Benchmark Velox |
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
What changes were proposed in this pull request?
Changes:
spark.gluten.sql.columnar.backend.velox.memInitCapacityto reduce memory arbitration times, default value is 8M.memoryPoolTransferCapacity_withspark.gluten.memory.reservationBlockSize, move reservation computation logic from SparkAllocationListener to Arbitrator.