Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.0, 3.1.1, 3.1.2, 3.2.0
-
None
-
None
Description
In Spark 3.1 we introduced shuffle migrations. However, it is possible that a shuffle file will still exist after it is no longer needed. I've only observed this in a back port branch with SQL, so I'll do some more digging.