[SPARK-23422] YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Assignee: Gabor Somogyi
• Reporter: Gabor Somogyi
• Created: 14/Feb/18 12:48
• Updated: 23/Feb/18 06:34
• Resolved: 15/Feb/18 11:53

Description

YarnShuffleIntegrationSuite fails when SPARK_PREPEND_CLASSES set to 1.

Normally mllib built before yarn module. When SPARK_PREPEND_CLASSES used mllib classes are on yarn test classpath.

Before 2.3 that did not cause issues. But 2.3 has SPARK-22450, which registered some mllib classes with the kryo serializer. Now it dies with the following error:

18/02/13 07:33:29 INFO SparkContext: Starting job: collect at YarnShuffleIntegrationSuite.scala:143
Exception in thread "dag-scheduler-event-loop" java.lang.NoClassDefFoundError: breeze/linalg/DenseMatrix

Issue Links

• links to: [Github] Pull Request #20608 (gaborgsomogyi)