[SPARK-1812] Support cross-building with Scala 2.11

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Build, Spark Core
• Assignee: Prashant Sharma
• Reporter: Matei Alexandru Zaharia
• Created: 12/May/14 23:53
• Updated: 17/Jan/15 14:26
• Resolved: 12/Nov/14 05:38

Description

Since Scala 2.10/2.11 are source compatible, we should be able to cross build for both versions. From what I understand there are basically three things we need to figure out:

1. Have a two versions of our dependency graph, one that uses 2.11 dependencies and the other that uses 2.10 dependencies.
2. Figure out how to publish different poms for 2.10 and 2.11.

I think (1) can be accomplished by having a scala 2.11 profile. (2) isn't really well supported by Maven since published pom's aren't generated dynamically. But we can probably script around it to make it work. I've done some initial sanity checks with a simple build here:

https://github.com/pwendell/scala-maven-crossbuild

Issue Links

• is duplicated by: SPARK-2040 [Duplicate] Support cross-building with Scala 2.11
• relates to: SPARK-4356 Test Scala 2.11 on Jenkins
• links to: [Github] Pull Request #996 (ScrapCodes)
• links to: [Github] Pull Request #1704 (avati)
• links to: [Github] Pull Request #2615 (ScrapCodes)
• links to: [Github] Pull Request #2947 (adampingel)
• links to: [Github] Pull Request #3111 (ScrapCodes)

Sub-Tasks

• update scalatest to version 2.1.5: Resolved
• update scala-logging-slf4j to version 2.1.2: Resolved
• Update breeze to version 0.9: Resolved
• Update akka to version 2.3.4: Resolved
• update json4s-jackson to version 3.2.10: Resolved
• update akka-zeromq to version 2.3.4: Resolved
• update kafka to version 0.8.2: Resolved
• update chill to version 0.5.0: Resolved
• update scala-maven-plugin to version 3.2.0: Resolved
• update algebird to 0.8.1: Resolved
• convert maven to archetype based build: Resolved
• Remove use of default arguments where disallowed by 2.11: Resolved
• Adjust build system to support building with scala 2.11 and fix tests.: Resolved
• Port repl to scala 2.11.: Resolved
• Adapt maven build to work without the need of hardcoding scala binary version in artifact id.: Resolved
• Adapt maven build to publish effective pom.: Resolved
• Rewrite the test for ActorInputStream.: Resolved
• Utils.resolveURI fails on upgrading scalaVersion to 2.11.2: Closed