New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix:read inconsistent when local file read handler init before flushing event to disk #1636
base: master
Are you sure you want to change the base?
Conversation
I don't see the root cause of this bug. @dingshun3016 Could you help add test case to simulate this or add more description |
I post one case log in our production env.
Second
Third
App log
|
Could you check the value of the metric Also I did not find the log you mentioned above "index files not find" in the latest code. Your code must be different. |
"index files not find" was added when I was troubleshooting the problem.It seem not same problem as #1620, after I added these codes, this case no longer occurred. |
What about the metric |
|
Can you provide a UT to reproduce this issue? That will be clear for us to figure out the root cause. |
Thanks for adding detailed logs, I have understood your point, the root cause is to partition shuffle reader handler is cached. But from my side, I don't think your code running in your producation env is the same with the current upstream master code. In current uniffle master code, if the indexFile is not found, it will throw exception, so the log may not the same. Based the above suppose, I think we just to handle the |
Please create the issue to track this bug @dingshun3016 |
Do we still need this? |
After thinking about it, I think throwing exception may cause call |
What changes were proposed in this pull request?
re init local file read handler when index file name is empty
Why are the changes needed?
some case will cause app to fail because of blocks read inconsistent
Does this PR introduce any user-facing change?
No.
How was this patch tested?
(Please test your changes, and provide instructions on how to test it: