[SPARK-26954] Do not attemp when user code throws exception

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 2.3.3, 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Deshan Xiao
• Created: 21/Feb/19 11:10
• Updated: 17/May/20 18:13
• Resolved: 21/Feb/19 17:39

Description

Yarn attemps the failed App depending on YarnRMClient#unregister. However, some attemps are useless like the example:

sc.parallelize(Seq(1,2,3)).map(_ => throw new RuntimeException("exception")).collect()

Also some attemps when "FileNorFoundException" is thrown in user code looks unreasonable.

Some environment errors, such as node dead, attemps reasonablely. So, it will be better that user exception will not attemp.

Issue Links

• links to: [Github] Pull Request #23857 (deshanxiao)
• links to: [Github] Pull Request #23857 (deshanxiao)
• links to: GitHub Pull Request #23857