Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Incomplete
-
3.4.0
-
None
-
None
Description
Ivy doesn't work correctly on IPv6.
SparkSubmitUtils.resolveMavenCoordinates(
"org.apache.logging.log4j:log4j-api:2.17.2",
SparkSubmitUtils.buildIvySettings(None, Some("/tmp/ivy")),
transitive = true)
% bin/spark-shell 22/06/16 22:22:12 WARN Utils: Your hostname, m1ipv6.local resolves to a loopback address: 127.0.0.1; using 2600:1700:232e:3de0:0:0:0:b instead (on interface en0) 22/06/16 22:22:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address ===== https://ipv6.repo1.maven.org/maven2/ =====https://maven-central.storage-download.googleapis.com/maven2/ Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 22/06/16 22:22:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://unknown1498776019fa.attlocal.net:4040 Spark context available as 'sc' (master = local[*], app id = local-1655443334687). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.4.0-SNAPSHOT /_/ Using Scala version 2.12.16 (OpenJDK 64-Bit Server VM, Java 17.0.3) Type in expressions to have them evaluated. Type :help for more information. scala> :paste -raw // Entering paste mode (ctrl-D to finish) package org.apache.spark.deploy object Download { SparkSubmitUtils.resolveMavenCoordinates( "org.apache.logging.log4j:log4j-api:2.17.2", SparkSubmitUtils.buildIvySettings(None, Some("/tmp/ivy")), transitive = true) } // Exiting paste mode, now interpreting. scala> org.apache.spark.deploy.Download ===== https://ipv6.repo1.maven.org/maven2/ =====https://maven-central.storage-download.googleapis.com/maven2/ :: loading settings :: url = jar:file:/Users/dongjoon/APACHE/spark/assembly/target/scala-2.12/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml Ivy Default Cache set to: /tmp/ivy/cache The jars for the packages stored in: /tmp/ivy/jars org.apache.logging.log4j#log4j-api added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent-f47b503f-897e-4b92-95da-3806c32c220f;1.0 confs: [default] :: resolution report :: resolve 95ms :: artifacts dl 0ms :: modules in use: --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 1 | 0 | 0 | 0 || 0 | 0 | --------------------------------------------------------------------- :: problems summary :: :::: WARNINGS module not found: org.apache.logging.log4j#log4j-api;2.17.2 ==== local-m2-cache: tried file:/Users/dongjoon/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar: file:/Users/dongjoon/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar ==== local-ivy-cache: tried /tmp/ivy/local/org.apache.logging.log4j/log4j-api/2.17.2/ivys/ivy.xml -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar: /tmp/ivy/local/org.apache.logging.log4j/log4j-api/2.17.2/jars/log4j-api.jar ==== ipv6: tried https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar: https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar ==== central: tried https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar: https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar ==== spark-packages: tried https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar: https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar :::::::::::::::::::::::::::::::::::::::::::::: :: UNRESOLVED DEPENDENCIES :: :::::::::::::::::::::::::::::::::::::::::::::: :: org.apache.logging.log4j#log4j-api;2.17.2: not found :::::::::::::::::::::::::::::::::::::::::::::: :::: ERRORS Server access error at url https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable) Server access error at url https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable) Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable) Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable) Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable) Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable) :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS java.lang.RuntimeException: [unresolved dependency: org.apache.logging.log4j#log4j-api;2.17.2: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1466) at org.apache.spark.deploy.Download$.<init>(<pastie>:4) at org.apache.spark.deploy.Download$.<clinit>(<pastie>) ... 47 elided