[SPARK-7726] Maven Install Breaks When Upgrading Scala 2.11.2-->[2.11.3 or higher]

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Build
• Target Version/s: 1.4.0
• Assignee: Dragos Dascalita Haut
• Reporter: Patrick Wendell
• Created: 19/May/15 09:21
• Updated: 11/Aug/15 05:39
• Resolved: 19/May/15 19:15

Description

This one took a long time to track down. The Maven install phase is part of our release process. It runs the "scala:doc" target to generate doc jars. Between Scala 2.11.2 and Scala 2.11.3, the behavior of this plugin changed in a way that breaks our build. In both cases, it returned an error (there has been a long running error here that we've always ignored), however in 2.11.3 that error became fatal and failed the entire build process. The upgrade occurred in SPARK-7092. Here is a simple reproduction:

./dev/change-version-to-2.11.sh
mvn clean install -pl network/common -pl network/shuffle -DskipTests -Dscala-2.11

This command exits success when Spark is at Scala 2.11.2 and fails with 2.11.3 or higher. In either case an error is printed:

[INFO] 
[INFO] --- scala-maven-plugin:3.2.0:doc-jar (attach-scaladocs) @ spark-network-shuffle_2.11 ---
/Users/pwendell/Documents/spark/network/shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/UploadBlock.java:56: error: not found: type Type
  protected Type type() { return Type.UPLOAD_BLOCK; }
            ^
/Users/pwendell/Documents/spark/network/shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/StreamHandle.java:37: error: not found: type Type
  protected Type type() { return Type.STREAM_HANDLE; }
            ^
/Users/pwendell/Documents/spark/network/shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/RegisterExecutor.java:44: error: not found: type Type
  protected Type type() { return Type.REGISTER_EXECUTOR; }
            ^
/Users/pwendell/Documents/spark/network/shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/OpenBlocks.java:40: error: not found: type Type
  protected Type type() { return Type.OPEN_BLOCKS; }
            ^
model contains 22 documentable templates
four errors found

Ideally we'd just dig in and fix this error. Unfortunately it's a very confusing error and I have no idea why it is appearing. I'd propose reverting SPARK-7092 in the mean time.

Issue Links

• is broken by: SPARK-7092 Update spark scala version to 2.11.6
• is duplicated by: SPARK-7670 Failure when building with scala 2.11 (after 1.3.1
• links to: [Github] Pull Request #6260 (dragos)
• links to: [Github] Pull Request #8095 (pwendell)