Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
1.5.2, 1.6.2, 2.0.0
-
None
Description
When I execute some sql statement periodically in the long running thriftserver, I found the disk device will be full after about one week later.
After check the file on linux, I found so many shuffle files left on the block-mgr dir whose shuffle stage had finished long time ago.
Finally I find when it's need to clean shuffle file, driver will total each executor to do the ShuffleClean. But when dynamic schedule is enabled, executor will be down itself and executor can't clean its shuffle file, then file was left.
I test it in Spark 1.5 but master branch must have this issue.
Attachments
Issue Links
- relates to
-
SPARK-26020 shuffle data from spark streaming not cleaned up when External Shuffle Service is enabled
- Resolved