Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

We found blockid was overflow when shuffle date is large enough, especially when the data is skewed.

For MR and TEZ, block is consist of sequentially increasing block id, task attempt id, partition id, task id.
The highest 12 bits are used for sequentially increasing block id, the next 6 bits are used for task attempt id, the next 24 bits are used for partition id, the lowest 21 bits are used for task id. (Note: Ignore highest bit)
So if block size in one partition is larger than 2^12, will be overflow.

For spark, block size supports maximum of 2^18. Because the lowest 21 bits is from TaskContext::taskAttemptId, this is a sequentially increasing number, can be used to identify tasks and task attempts.

I think we could move the task attempt id to the lowest 21 bits, reduce the number of bits allocated to task attempt id. In general task attempt id is not large.

How should we improve?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!