Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21507

Exception when using spark.jars.packages

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:

      Description

      When more than one process is using packages option it's possible to create following exception

      INFO - Ivy Default Cache set to: /home/bi/.ivy2/cache
      INFO - The jars for the packages stored in: /home/bi/.ivy2/jars
      INFO - :: loading settings :: url = jar:file:/opt/spark-2.2.0/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
      INFO - datastax#spark-cassandra-connector added as a dependency
      INFO - :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
      INFO - confs: [default]
      INFO - found datastax#spark-cassandra-connector;2.0.3-s_2.11 in spark-packages
      INFO - found commons-beanutils#commons-beanutils;1.9.3 in central
      INFO - found commons-collections#commons-collections;3.2.2 in local-m2-cache
      INFO - found joda-time#joda-time;2.3 in central
      INFO - found org.joda#joda-convert;1.2 in central
      INFO - found com.twitter#jsr166e;1.1.0 in central
      INFO - found io.netty#netty-all;4.0.33.Final in central
      INFO - found org.scala-lang#scala-reflect;2.11.8 in local-m2-cache
      INFO - :: resolution report :: resolve 1042ms :: artifacts dl 51ms
      INFO - :: modules in use:
      INFO - com.twitter#jsr166e;1.1.0 from central in [default]
      INFO - commons-beanutils#commons-beanutils;1.9.3 from central in [default]
      INFO - commons-collections#commons-collections;3.2.2 from local-m2-cache in [default]
      INFO - datastax#spark-cassandra-connector;2.0.3-s_2.11 from spark-packages in [default]
      INFO - io.netty#netty-all;4.0.33.Final from central in [default]
      INFO - joda-time#joda-time;2.3 from central in [default]
      INFO - org.joda#joda-convert;1.2 from central in [default]
      INFO - org.scala-lang#scala-reflect;2.11.8 from local-m2-cache in [default]
      INFO - ---------------------------------------------------------------------
      INFO - |                  |            modules            ||   artifacts   |
      INFO - |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
      INFO - ---------------------------------------------------------------------
      INFO - |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
      INFO - ---------------------------------------------------------------------
      INFO - :: retrieving :: org.apache.spark#spark-submit-parent
      INFO - confs: [default]
      INFO - Exception in thread "main" java.lang.RuntimeException: problem during retrieve of org.apache.spark#spark-submit-parent: java.text.ParseException: failed to parse report: /home/bi/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml: Premature end of file.
      INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:249)
      INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:83)
      INFO - at org.apache.ivy.Ivy.retrieve(Ivy.java:551)
      INFO - at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1180)
      INFO - at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:298)
      INFO - at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
      INFO - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
      INFO - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      INFO - Caused by: java.text.ParseException: failed to parse report: /home/bi/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml: Premature end of file.
      INFO - at org.apache.ivy.plugins.report.XmlReportParser.parse(XmlReportParser.java:293)
      INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.determineArtifactsToCopy(RetrieveEngine.java:329)
      INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:118)
      INFO - ... 7 more
      INFO - Caused by: org.xml.sax.SAXParseException; Premature end of file.
      INFO - at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
      INFO - at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
      INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
      INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
      INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
      INFO - at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
      INFO - at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
      INFO - at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
      INFO - at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
      INFO - at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
      INFO - at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
      INFO - at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
      INFO - at javax.xml.parsers.SAXParser.parse(SAXParser.java:328)
      INFO - at org.apache.ivy.plugins.report.XmlReportParser$SaxXmlReportParser.parse(XmlReportParser.java:249)
      INFO - at org.apache.ivy.plugins.report.XmlReportParser.parse(XmlReportParser.java:291)
      INFO - ... 9 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                maver1ck Maciej BryƄski
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: