Details
-
New Feature
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
3.2.0
-
None
-
None
Description
Shuffle data corruption is a long-standing issue in Spark. For example, in SPARK-18105, people continually reports corruption issue. However, data corruption is difficult to reproduce in most cases and even harder to tell the root cause. We don't know if it's a Spark issue or not. With the checksum support for the shuffle, Spark itself can at least distinguish the cause between disk and network, which is very important for users.
Attachments
Issue Links
1.
|
Write checksum files for shuffle |
|
Resolved | wuyi |
2.
|
Diagnose shuffle data corruption by checksum |
|
Resolved | wuyi |
3.
|
Add shuffle checksum support for push-based shuffle |
|
Open | Unassigned |
4.
|
Add documentation for shuffle checksum |
|
Resolved | wuyi |
5.
|
Skip diagnosis ob merged blocks from push-based shuffle |
|
Resolved | Cheng Pan |