Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2338

Jenkins Spark-Master-Maven-with-YARN builds failing due to test misconfiguration

    Details

      Description

      https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.2.0,label=centos/
      https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.3.0,label=centos/

      These builds are currently failing due to the builder configuration being incomplete. After building, they specify the test command as:

      /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn -Dhadoop.version=2.3.0 -Dlabel=centos test -Pyarn -Phive
      

      However, it is not enough to specify the hadoop.version, the tests should instead be run using the hadoop-2.2 and hadoop-2.3 profiles respectively.

      For example:

      /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn -Phadoop2.2 -Dlabel=centos test -Pyarn -Phive
      

      These profiles will not only set the appropriate hadoop.version but also set the version of protobuf-java required by yarn (2.5.0). Without the correct profile set, the test run fails at:

      *** RUN ABORTED ***
        java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
      

      since it is getting the default version of protobuf-java (2.4.1) which has the old incompatible version of getUnknownFields.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pmackinn Pete MacKinnon
                Reporter:
                pmackinn Pete MacKinnon
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: