[SPARK-12807] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.1, 2.0.0
• Component/s: Shuffle, Spark Core, YARN

Environment

A Hadoop cluster with Jackson 2.2.3, spark running with dynamic allocation enabled

• Target Version/s: 1.6.1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 13/Jan/16 16:38
• Updated: 17/May/20 18:17
• Resolved: 09/Feb/16 19:02

Description

When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, you get to see a stack trace in the NM logs, indicating a jackson 2.x version mismatch.

(reported on the spark dev list)

Issue Links

• depends upon: HADOOP-12705 Upgrade Jackson 2.2.3 to 2.7.8
• depends upon: YARN-1593 support out-of-proc AuxiliaryServices
• depends upon: YARN-4577 Enable aux services to have their own custom classpath/jar file
• is broken by: SPARK-9439 ExternalShuffleService should be robust to NodeManager restarts in yarn
• links to: [Github] Pull Request #10780 (steveloughran)
• links to: [Github] Pull Request #10782 (steveloughran)