[SPARK-37097] yarn-cluster mode, unregister timeout cause spark retry but AM container exit with code 0

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: angerszhu
• Created: 22/Oct/21 07:46
• Updated: 22/Oct/21 09:44

Description

1. Cluster mode AM shutdown hook triggered
2. am unregister from RM timeout, but AM shutdown hook have try catch, so AM container exit with code 0.
3. Since RM lose connection with AM, then treat this container as failed.
4. Then client side got application report as final status failed but am container exit code 0. Then retry.

Issue Links

• links to: [Github] Pull Request #34366 (AngersZhuuuu)