Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2338

Jenkins Spark-Master-Maven-with-YARN builds failing due to test misconfiguration

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.2.0,label=centos/
      https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.3.0,label=centos/

      These builds are currently failing due to the builder configuration being incomplete. After building, they specify the test command as:

      /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn -Dhadoop.version=2.3.0 -Dlabel=centos test -Pyarn -Phive
      

      However, it is not enough to specify the hadoop.version, the tests should instead be run using the hadoop-2.2 and hadoop-2.3 profiles respectively.

      For example:

      /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn -Phadoop2.2 -Dlabel=centos test -Pyarn -Phive
      

      These profiles will not only set the appropriate hadoop.version but also set the version of protobuf-java required by yarn (2.5.0). Without the correct profile set, the test run fails at:

      *** RUN ABORTED ***
        java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
      

      since it is getting the default version of protobuf-java (2.4.1) which has the old incompatible version of getUnknownFields.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            pmackinn Pete MacKinnon
            pmackinn Pete MacKinnon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment