Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39457 Support IPv6-only environment
  3. SPARK-39500

Ivy doesn't work correctly on IPv6-only environment

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Incomplete
    • 3.4.0
    • None
    • Spark Core
    • None

    Description

      Ivy doesn't work correctly on IPv6.

        SparkSubmitUtils.resolveMavenCoordinates(
          "org.apache.logging.log4j:log4j-api:2.17.2",
          SparkSubmitUtils.buildIvySettings(None, Some("/tmp/ivy")),
          transitive = true)
      
      % bin/spark-shell
      22/06/16 22:22:12 WARN Utils: Your hostname, m1ipv6.local resolves to a loopback address: 127.0.0.1; using 2600:1700:232e:3de0:0:0:0:b instead (on interface en0)
      22/06/16 22:22:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
      ===== https://ipv6.repo1.maven.org/maven2/
      =====https://maven-central.storage-download.googleapis.com/maven2/
      Setting default log level to "WARN".
      To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
      22/06/16 22:22:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Spark context Web UI available at http://unknown1498776019fa.attlocal.net:4040
      Spark context available as 'sc' (master = local[*], app id = local-1655443334687).
      Spark session available as 'spark'.
      Welcome to
            ____              __
           / __/__  ___ _____/ /__
          _\ \/ _ \/ _ `/ __/  '_/
         /___/ .__/\_,_/_/ /_/\_\   version 3.4.0-SNAPSHOT
            /_/
      
      Using Scala version 2.12.16 (OpenJDK 64-Bit Server VM, Java 17.0.3)
      Type in expressions to have them evaluated.
      Type :help for more information.
      
      scala> :paste -raw
      // Entering paste mode (ctrl-D to finish)
      
      package org.apache.spark.deploy
      
      object Download {
        SparkSubmitUtils.resolveMavenCoordinates(
          "org.apache.logging.log4j:log4j-api:2.17.2",
          SparkSubmitUtils.buildIvySettings(None, Some("/tmp/ivy")),
          transitive = true)
      }
      
      // Exiting paste mode, now interpreting.
      
      
      scala> org.apache.spark.deploy.Download
      ===== https://ipv6.repo1.maven.org/maven2/
      =====https://maven-central.storage-download.googleapis.com/maven2/
      :: loading settings :: url = jar:file:/Users/dongjoon/APACHE/spark/assembly/target/scala-2.12/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
      Ivy Default Cache set to: /tmp/ivy/cache
      The jars for the packages stored in: /tmp/ivy/jars
      org.apache.logging.log4j#log4j-api added as a dependency
      :: resolving dependencies :: org.apache.spark#spark-submit-parent-f47b503f-897e-4b92-95da-3806c32c220f;1.0
              confs: [default]
      :: resolution report :: resolve 95ms :: artifacts dl 0ms
              :: modules in use:
              ---------------------------------------------------------------------
              |                  |            modules            ||   artifacts   |
              |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
              ---------------------------------------------------------------------
              |      default     |   1   |   0   |   0   |   0   ||   0   |   0   |
              ---------------------------------------------------------------------
      
      :: problems summary ::
      :::: WARNINGS
                      module not found: org.apache.logging.log4j#log4j-api;2.17.2
      
              ==== local-m2-cache: tried
      
                file:/Users/dongjoon/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
      
                -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
      
                file:/Users/dongjoon/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar
      
              ==== local-ivy-cache: tried
      
                /tmp/ivy/local/org.apache.logging.log4j/log4j-api/2.17.2/ivys/ivy.xml
      
                -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
      
                /tmp/ivy/local/org.apache.logging.log4j/log4j-api/2.17.2/jars/log4j-api.jar
      
              ==== ipv6: tried
      
                https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
      
                -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
      
                https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar
      
              ==== central: tried
      
                https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
      
                -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
      
                https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar
      
              ==== spark-packages: tried
      
                https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
      
                -- artifact org.apache.logging.log4j#log4j-api;2.17.2!log4j-api.jar:
      
                https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar
      
                      ::::::::::::::::::::::::::::::::::::::::::::::
      
                      ::          UNRESOLVED DEPENDENCIES         ::
      
                      ::::::::::::::::::::::::::::::::::::::::::::::
      
                      :: org.apache.logging.log4j#log4j-api;2.17.2: not found
      
                      ::::::::::::::::::::::::::::::::::::::::::::::
      
      
      :::: ERRORS
              Server access error at url https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable)
      
              Server access error at url https://ipv6.repo1.maven.org/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable)
      
              Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable)
      
              Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable)
      
              Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (java.net.SocketException: Network is unreachable)
      
              Server access error at url https://maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar (java.net.SocketException: Network is unreachable)
      
      
      :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
      java.lang.RuntimeException: [unresolved dependency: org.apache.logging.log4j#log4j-api;2.17.2: not found]
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1466)
        at org.apache.spark.deploy.Download$.<init>(<pastie>:4)
        at org.apache.spark.deploy.Download$.<clinit>(<pastie>)
        ... 47 elided
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment