[SPARK-21369] Don't use Scala classes in external shuffle service

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: Shuffle, Spark Core, YARN
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 10/Jul/17 20:44
• Updated: 17/May/20 18:13
• Resolved: 11/Jul/17 03:27

Description

Right now the external shuffle service uses Scala Tuple2. However, the Scala library won't be shaded into the yarn shuffle assembly jar. Then when the codes are called, it will throw ClassNotFoundException.

Right now it's safe because we disabled spark.reducer.maxReqSizeShuffleToMem by default. However, to allow using spark.reducer.maxReqSizeShuffleToMem for Yarn users, we should remove all usages of Tuples.

Issue Links

• links to: [Github] Pull Request #18593 (zsxwing)