Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2075

Anonymous classes are missing from Spark distribution

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.2.1, 1.3.0
    • Component/s: Build, Spark Core
    • Labels:
      None

      Description

      Running a job built against the Maven dep for 1.0.0 and the hadoop1 distribution produces:

      java.lang.ClassNotFoundException:
      org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1
      

      Here's what's in the Maven dep as of 1.0.0:

      jar tvf ~/.m2/repository/org/apache/spark/spark-core_2.10/1.0.0/spark-core_2.10-1.0.0.jar | grep 'rdd/RDD' | grep 'saveAs'
        1519 Mon May 26 13:57:58 PDT 2014 org/apache/spark/rdd/RDD$anonfun$saveAsTextFile$1.class
        1560 Mon May 26 13:57:58 PDT 2014 org/apache/spark/rdd/RDD$anonfun$saveAsTextFile$2.class
      

      And here's what's in the hadoop1 distribution:

      jar tvf spark-assembly-1.0.0-hadoop1.0.4.jar| grep 'rdd/RDD' | grep 'saveAs'
      

      I.e., it's not there. It is in the hadoop2 distribution:

      jar tvf spark-assembly-1.0.0-hadoop2.2.0.jar| grep 'rdd/RDD' | grep 'saveAs'
        1519 Mon May 26 07:29:54 PDT 2014 org/apache/spark/rdd/RDD$anonfun$saveAsTextFile$1.class
        1560 Mon May 26 07:29:54 PDT 2014 org/apache/spark/rdd/RDD$anonfun$saveAsTextFile$2.class
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zsxwing Shixiong Zhu
                Reporter:
                paulrbrown Paul R. Brown
              • Votes:
                2 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: