ok not that MAHOUT-2020 is resolved, I looked at the scopt issue and found:
- all the correct scopt artifact exist in remote repos for all scala versions and they are being found by the mahout build.
- the ids for artifact etc are correct as per ^^^
- I checked all the tagged versions of Mahout back to 12.0. Not sure when the drivers stopped working but there has been no change to any reference to scopt in any POM. And since people have been using it and asking questions on the mailing list I will assume that up till the last build changes the drivers worked.
- The vienna-cl and java to c bindings are in the assembly pom so these classes are getting to the Spark Executors.
- I've checked compute-classpath.sh and the mahout script where changes were small and not relevant.
- I've looked at the contents of the mahout*dependency-reduced.jar, which should have the things listed below and it does not, in only had guava, apache commons and fastutils. It is supposed to have:
This all leads me to believe that something in the build no longer makes that dependency-reduced.jar available to the Java Driver code since those other libs in the assembly are probably all hadoop or Spark Executor code, not needed in the Mahout driver. This is likely to have been a side effect of the build refactoring
Trevor Grant does "dependencies-reduced.jar" which contains Scopt get its scala.compat.version fixed? It seems like the jar is missing anything with scala.compat.version but this may be a red herring.