Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20815

NullPointerException in RPackageUtils#checkManifestForR

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.2.0, 2.3.0
    • Component/s: SparkR
    • Labels:
      None

      Description

      Some jars don't have manifest files in them, such as in my case javax.inject-1.jar and value-2.2.1-annotations.jar

      This causes the below NPE:

      Exception in thread "main" java.lang.NullPointerException
              at org.apache.spark.deploy.RPackageUtils$.checkManifestForR(RPackageUtils.scala:95)
              at org.apache.spark.deploy.RPackageUtils$$anonfun$checkAndBuildRPackage$1$$anonfun$apply$1.apply$mcV$sp(RPackageUtils.scala:180)
              at org.apache.spark.deploy.RPackageUtils$$anonfun$checkAndBuildRPackage$1$$anonfun$apply$1.apply(RPackageUtils.scala:180)
              at org.apache.spark.deploy.RPackageUtils$$anonfun$checkAndBuildRPackage$1$$anonfun$apply$1.apply(RPackageUtils.scala:180)
              at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1322)
              at org.apache.spark.deploy.RPackageUtils$$anonfun$checkAndBuildRPackage$1.apply(RPackageUtils.scala:202)
              at org.apache.spark.deploy.RPackageUtils$$anonfun$checkAndBuildRPackage$1.apply(RPackageUtils.scala:175)
              at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
              at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
              at org.apache.spark.deploy.RPackageUtils$.checkAndBuildRPackage(RPackageUtils.scala:175)
              at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:311)
              at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:152)
              at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:118)
              at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      

      due to RPackageUtils#checkManifestForR assuming jar.getManifest is non-null.

      However per the JDK spec it can be null:

          /**
           * Returns the jar file manifest, or <code>null</code> if none.
           *
           * @return the jar file manifest, or <code>null</code> if none
           *
           * @throws IllegalStateException
           *         may be thrown if the jar file has been closed
           * @throws IOException  if an I/O error has occurred
           */
          public Manifest getManifest() throws IOException {
              return getManifestFromReference();
          }
      

      This method should do a null check and return false if the manifest is null (meaning no R code in that jar)

        Attachments

          Activity

            People

            • Assignee:
              jrshust James Shuster
              Reporter:
              aash Andrew Ash
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: