[SPARK-43510] Spark application hangs when YarnAllocator adds running executors after processing completed containers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.1, 3.5.0
• Component/s: YARN
• Assignee: Manu Zhang
• Reporter: Manu Zhang
• Created: 15/May/23 15:35
• Updated: 06/Jun/23 13:30
• Resolved: 06/Jun/23 13:30

Description

I see application hangs when containers are preempted immediately after allocation as follows.

23/05/14 09:11:33 INFO YarnAllocator: Launching container container_e3812_1684033797982_57865_01_000382 on host hdc42-mcc10-01-0910-4207-015-tess0028.stratus.rno.ebay.com for executor with ID 277 for ResourceProfile Id 0 
23/05/14 09:11:33 WARN YarnAllocator: Cannot find executorId for container: container_e3812_1684033797982_57865_01_000382
23/05/14 09:11:33 INFO YarnAllocator: Completed container container_e3812_1684033797982_57865_01_000382 (state: COMPLETE, exit status: -102)
23/05/14 09:11:33 INFO YarnAllocator: Container container_e3812_1684033797982_57865_01_000382 was preempted.

Note the warning log where YarnAllocator cannot find executorId for the container when processing completed containers. The only plausible cause is YarnAllocator added the running executor after processing completed containers. The former happens in a separate thread after executor launch.

YarnAllocator believes there are still running executors, although they are already lost due to preemption. Hence, the application hangs without any running executors.

Issue Links

• links to: [Github] Pull Request #41173 (manuzhang)