-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-19180. EC: Fix calculation errors caused by special index order #6813
base: trunk
Are you sure you want to change the base?
Conversation
@zhangshuyan0 @haiyang1987 Could you help review this PR? I'm not very familiar with EC, but I've noticed that you have submitted quite a few improvements related to EC. Thank you very much! @zhengchenyu Thank for the contribution! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zhengchenyu Thanks for your report! If there is a parity index
smaller than data index
, there will be a bug in decoding calculation. Is this what you mean?
Firstly, I am curious when parity index
will be smaller?
Secondly, if this situation occurs, the reading of EC files may also be affected. So, I think we won't actively modify the code to make this happen.
@zhangshuyan0 |
@zhengchenyu Thanks for your explanation! I got it. This PR LGTM. |
HDFS-15186 has described this problem, but does not fundamentally solve it. |
@zhengchenyu Thanks for your reply. I agree. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make sense.
Maybe you should move this ticket to HADOOP and move the UT to common.
@ZanderXu |
you can move hdfs to common |
@haiyang1987 thank you for your reminder! |
💔 -1 overall
This message was automatically generated. |
The reported error "Doesn't support SM4 CTR." seems to be not related to this PR. Perhaps the compiler environment has changed? |
@ZanderXu Can you please review this PR? |
Description of PR
I found that if the erasedIndexes distribution is such that the parity index is in front of the data index, ec will produce wrong results when decoding.
In fact, HDFS-15186 has described this problem, but does not fundamentally solve it.
The reason is that the code assumes that erasedIndexes is preceded by the data index and followed by parity index. If there is a parity index placed in front of the data index in the incoming code, a calculation error will occur.
How was this patch tested?
The TestErasureCodingEncodeAndDecode unit test and the erasure_code_test binary were executed on different machines. The test machines include those with isa-l installed and those without isa-l installed.
For code changes: