[SPARK-2338] Jenkins Spark-Master-Maven-with-YARN builds failing due to test misconfiguration

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Component/s: Build, Project Infra, YARN
• Labels: hadoop2 jenkins maven protobuf yarn

Environment

https://amplab.cs.berkeley.edu/jenkins

• Assignee: Pete MacKinnon
• Reporter: Pete MacKinnon
• Created: 01/Jul/14 19:16
• Updated: 09/Jul/14 16:55
• Resolved: 09/Jul/14 11:36

Description

https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.2.0,label=centos/
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/hadoop.version=2.3.0,label=centos/

These builds are currently failing due to the builder configuration being incomplete. After building, they specify the test command as:

/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn -Dhadoop.version=2.3.0 -Dlabel=centos test -Pyarn -Phive

However, it is not enough to specify the hadoop.version, the tests should instead be run using the hadoop-2.2 and hadoop-2.3 profiles respectively.

For example:

/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn -Phadoop2.2 -Dlabel=centos test -Pyarn -Phive

These profiles will not only set the appropriate hadoop.version but also set the version of protobuf-java required by yarn (2.5.0). Without the correct profile set, the test run fails at:

*** RUN ABORTED ***
  java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;

since it is getting the default version of protobuf-java (2.4.1) which has the old incompatible version of getUnknownFields.

Issue Links

• is part of: SPARK-2232 Fix Jenkins tests in Maven