[SPARK-39491] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: Yang Jie
• Reporter: Thomas Graves
• Created: 16/Jun/22 15:37
• Updated: 12/Dec/22 18:10
• Resolved: 20/Jun/22 08:31

Description

trying to build with the hadoop-2 profile, which uses hadoop 2.7 version fails due to:

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454: value DECOMMISSIONING is not a member of object org.apache.hadoop.yarn.api.records.NodeState

 

DECOMMISSIONING was only added in hadoop 2.8.

This was added in https://issues.apache.org/jira/browse/SPARK-30835

Issue Links

• is caused by: SPARK-30835 Add support for YARN decommissioning & pre-emption
• links to: [Github] Pull Request #36890 (abhishekd0907)
• links to: [Github] Pull Request #36890 (abhishekd0907)
• links to: [Github] Pull Request #36917 (LuciferYang)