[SPARK-2075] Anonymous classes are missing from Spark distribution

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.1, 1.3.0
• Component/s: Build, Spark Core
• Assignee: Shixiong Zhu
• Reporter: Paul R. Brown
• Created: 08/Jun/14 19:44
• Updated: 22/Dec/14 10:12
• Resolved: 22/Dec/14 06:10

Description

Running a job built against the Maven dep for 1.0.0 and the hadoop1 distribution produces:

java.lang.ClassNotFoundException:
org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1

Here's what's in the Maven dep as of 1.0.0:

jar tvf ~/.m2/repository/org/apache/spark/spark-core_2.10/1.0.0/spark-core_2.10-1.0.0.jar | grep 'rdd/RDD' | grep 'saveAs'
  1519 Mon May 26 13:57:58 PDT 2014 org/apache/spark/rdd/RDD$anonfun$saveAsTextFile$1.class
  1560 Mon May 26 13:57:58 PDT 2014 org/apache/spark/rdd/RDD$anonfun$saveAsTextFile$2.class

And here's what's in the hadoop1 distribution:

jar tvf spark-assembly-1.0.0-hadoop1.0.4.jar| grep 'rdd/RDD' | grep 'saveAs'

I.e., it's not there. It is in the hadoop2 distribution:

jar tvf spark-assembly-1.0.0-hadoop2.2.0.jar| grep 'rdd/RDD' | grep 'saveAs'
  1519 Mon May 26 07:29:54 PDT 2014 org/apache/spark/rdd/RDD$anonfun$saveAsTextFile$1.class
  1560 Mon May 26 07:29:54 PDT 2014 org/apache/spark/rdd/RDD$anonfun$saveAsTextFile$2.class

Issue Links

• is related to: SPARK-2292 NullPointerException in JavaPairRDD.mapToPair
• links to: [Github] Pull Request #3740 (zsxwing)
• links to: [Github] Pull Request #3758 (zsxwing)