In order to upgrade to Spark 1.5.0 once it is released (~Sep 2015), we will need to upgrade the Bigtop toolchain to Maven 3.3.3 due to https://issues.apache.org/jira/browse/SPARK-9521.
Upgrade Mahout to 0.11.0
Upgrade Tez version to 0.6.2
Require Maven 3.3.3+ in the build
While working on BIGTOP-1795 I tried maven 3.3.3.
Compiling tez with 3.3.3 this is blocked by MNG-5787 aka MNG-5835.
We cannot switch all components until both problems are resolved
Ah, yes, I'd forgotten about that. What exactly is the error you get when building Tez with Maven 3.3.3? Is there any way that the problem can be solved within the Tez project, or does it actually need to be solved in Maven with these two MNG JIRAs?
TEZ-2560 blocked upgrading Maven 3.3+
But Tez 6.2 will include TEZ-2560 and the Tez community will release 6.2 soon. I'll file a JIRA for upgrading Tez. Thanks!
Tez 0.6.2 is out now! I filed BIGTOP-1964 and attached a patch.
Patch (we really should reengineer bigtop_toolchain, its a mess)
+1. Tested by running a full build against new maven version. The 3.3 works fine with tez 0.6.2. Please commit or I'll do it later. Thanks.
applied and resolved. Thanks Evans Ye
Can you create new "official" docker containers with current toolchain already applied ?
Sure. Actually I've created new images: http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain-Trunk/
But a thing came up into my mind is that how do we named these images?
Right now we don't tag images with the release number, which I admit it isn't a good practice:
I think we can name them as the following scheme:
Or probably you have a better design in BIGTOP-1996, then we can discuss there.