just an FYI, this broke my own unit tests on the SPARK-7481 branch
2016-10-13 18:51:13,888 [ScalaTest-main] INFO cloud.CloudSuite (Logging.scala:logInfo(54)) - Loading configuration from ../../cloud.xml
2016-10-13 18:51:14,214 [ScalaTest-main] WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
*** RUN ABORTED ***
[INFO] BUILD FAILURE
But this problem goes away in spark-assembly, the release of spark, etc. Purely this module. Which is why I didn't catch this earlier as the system integration tests were all happy.
- there's a newer version of jackson in use in spark (2.6.5)
- which overrides the declarations of jackson-annotations and jackson-databind under hadoop-aws
- and which have transitive dependencies on jackson-common.
- the explicitdeclaration of jackson-common has pulled that reference one step up the dependency graph (i.e. from under spark-cloud/hadoop-aws/amazon-aws/jackson-common.jar) to spark-cloud/hadoop-aws/jackson-common.jar.
- which gives the hadoop-aws version precedence over the one transitively referenced by the (overridde) jackson-annotations, pulled in directly from spark-core JAR.
- so creating a version inconsistency which surfaces during test runs.
The problem isn't in spark-assembly.jar as it refers to spark-core jar directly, plcking that version up instead.
Essentially: the fact that maven uses closest-version first in its version resolution policy means that the depth of transitive dependencies controls whether things run or not; the explicit declaration of the dependency was enough to cause this to surface.
Fix: explicitly exclude the hadoop-aws jackson dependencies, as was already done for hadoop-azure.
This is not me faulting my own work (how would I!), only showing that you do need to be careful across projects as to what transitive stuff you pull in, as it turns out to be incredibly brittle. We didn't change the jackson version here, only made that choice explicit, and a downstream test suite fails.