[SPARK-31582] Being able to not populate Hadoop classpath

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.5
• Fix Version/s: 2.4.6, 3.0.0
• Component/s: Spark Core, YARN
• Assignee: DB Tsai
• Reporter: DB Tsai
• Created: 27/Apr/20 21:01
• Updated: 17/May/20 18:14
• Resolved: 29/Apr/20 21:11

Description

Spark Yarn client will populate hadoop classpath from `yarn.application.classpath` and ``mapreduce.application.classpath`. However, for Spark with embedded hadoop build, it will result jar conflicts because spark distribution can contain different version of hadoop jars.

We are adding a new Yarn configuration to not populate hadoop classpath from  `yarn.application.classpath` and ``mapreduce.application.classpath`.

Issue Links

• links to: [Github] Pull Request #28376 (dbtsai)
• links to: [Github] Pull Request #28376 (dbtsai)

Sub-Tasks

• Only populate Hadoop classpath for no-hadoop build: Resolved