[SPARK-21507] Exception when using spark.jars.packages - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Minor
Resolution: Incomplete
Affects Version/s: 2.2.0
Fix Version/s: None
Component/s: Spark Core
Labels:
- bulk-closed

Description

When more than one process is using packages option it's possible to create following exception

INFO - Ivy Default Cache set to: /home/bi/.ivy2/cache
INFO - The jars for the packages stored in: /home/bi/.ivy2/jars
INFO - :: loading settings :: url = jar:file:/opt/spark-2.2.0/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
INFO - datastax#spark-cassandra-connector added as a dependency
INFO - :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
INFO - confs: [default]
INFO - found datastax#spark-cassandra-connector;2.0.3-s_2.11 in spark-packages
INFO - found commons-beanutils#commons-beanutils;1.9.3 in central
INFO - found commons-collections#commons-collections;3.2.2 in local-m2-cache
INFO - found joda-time#joda-time;2.3 in central
INFO - found org.joda#joda-convert;1.2 in central
INFO - found com.twitter#jsr166e;1.1.0 in central
INFO - found io.netty#netty-all;4.0.33.Final in central
INFO - found org.scala-lang#scala-reflect;2.11.8 in local-m2-cache
INFO - :: resolution report :: resolve 1042ms :: artifacts dl 51ms
INFO - :: modules in use:
INFO - com.twitter#jsr166e;1.1.0 from central in [default]
INFO - commons-beanutils#commons-beanutils;1.9.3 from central in [default]
INFO - commons-collections#commons-collections;3.2.2 from local-m2-cache in [default]
INFO - datastax#spark-cassandra-connector;2.0.3-s_2.11 from spark-packages in [default]
INFO - io.netty#netty-all;4.0.33.Final from central in [default]
INFO - joda-time#joda-time;2.3 from central in [default]
INFO - org.joda#joda-convert;1.2 from central in [default]
INFO - org.scala-lang#scala-reflect;2.11.8 from local-m2-cache in [default]
INFO - ---------------------------------------------------------------------
INFO - |                  |            modules            ||   artifacts   |
INFO - |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
INFO - ---------------------------------------------------------------------
INFO - |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
INFO - ---------------------------------------------------------------------
INFO - :: retrieving :: org.apache.spark#spark-submit-parent
INFO - confs: [default]
INFO - Exception in thread "main" java.lang.RuntimeException: problem during retrieve of org.apache.spark#spark-submit-parent: java.text.ParseException: failed to parse report: /home/bi/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml: Premature end of file.
INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:249)
INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:83)
INFO - at org.apache.ivy.Ivy.retrieve(Ivy.java:551)
INFO - at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1180)
INFO - at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:298)
INFO - at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
INFO - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
INFO - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
INFO - Caused by: java.text.ParseException: failed to parse report: /home/bi/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml: Premature end of file.
INFO - at org.apache.ivy.plugins.report.XmlReportParser.parse(XmlReportParser.java:293)
INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.determineArtifactsToCopy(RetrieveEngine.java:329)
INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:118)
INFO - ... 7 more
INFO - Caused by: org.xml.sax.SAXParseException; Premature end of file.
INFO - at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
INFO - at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
INFO - at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
INFO - at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
INFO - at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
INFO - at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
INFO - at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
INFO - at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
INFO - at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
INFO - at javax.xml.parsers.SAXParser.parse(SAXParser.java:328)
INFO - at org.apache.ivy.plugins.report.XmlReportParser$SaxXmlReportParser.parse(XmlReportParser.java:249)
INFO - at org.apache.ivy.plugins.report.XmlReportParser.parse(XmlReportParser.java:291)
INFO - ... 9 more