[SPARK-11798] Datanucleus jars is missing under lib_managed/jars

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: Build, SQL
• Assignee: Unassigned
• Reporter: Jeff Zhang
• Created: 18/Nov/15 02:07
• Updated: 14/Mar/17 15:34
• Resolved: 08/Jan/16 01:34

Description

I notice the comments in https://github.com/apache/spark/pull/9575 said that Datanucleus related jars will still be copied to lib_managed/jars. But I don't see any jars under lib_managed/jars. The weird thing is that I see the jars on another machine, but could not see jars on my laptop even after I delete the whole spark project and start from scratch. Does it related with environments ? I try to add the following code in SparkBuild.scala to track the issue, it shows that the jars is empty.

deployDatanucleusJars := {
      val jars: Seq[File] = (fullClasspath in assembly).value.map(_.data)
        .filter(_.getPath.contains("org.datanucleus"))
      // this is what I added
      println("*********************************************")
      println("fullClasspath:"+fullClasspath)
      println("assembly:"+assembly)
      println("jars:"+jars.map(_.getAbsolutePath()).mkString(","))
      //

Issue Links

• depends upon: SPARK-7841 Spark build should not use lib_managed for dependencies