[SPARK-41168] Spark Master OOM when Worker No space left on device

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.3.1
• Component/s: Deploy
• Assignee: Unassigned
• Reporter: pin_zhang
• Created: 17/Nov/22 04:29
• Updated: 12/Dec/22 18:10
• Resolved: 17/Nov/22 11:03

Description

Spark master + 1 Spark workers (One is No space left on device)
1. Submit a app with 2 instances
2. Stop the good worker
3. Spark master launch executor continously
Bad Spark worker cannot create folder.
Result in large number of executors kept in spark master memory
2022-11-15 22:21:43 INFO Master:54 - Launching executor app-20221115221952-0016/93441 on worker worker-20221115202400-10.111.1.10-40011
2022-11-15 22:21:43 INFO Master:54 - Removing executor app-20221115221952-0016/93441 because it is FAILED
2022-11-15 22:21:43 INFO Master:54 - Launching executor app-20221115221952-0016/93442 on worker worker-20221115202400-10.111.1.10-40011
2022-11-15 22:21:43 INFO Master:54 - Removing executor app-20221115221952-0016/93442 because it is FAILED