Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27550

Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.3
    • Fix Version/s: 2.4.3
    • Component/s: Tests
    • Labels:
      None

      Description

      Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more.

      Kakfa 0.8 doesn't have Scala-2.12 artifacts, e.g., `org.apache.kafka:kafka_2.12:jar:0.8.2.1`. This issue aims to fix `test-dependencies.sh` script to understand Scala binary version.

      $ dev/change-scala-version.sh 2.12
      $ dev/test-dependencies.sh
      Using `mvn` from path: /usr/local/bin/mvn
      Using `mvn` from path: /usr/local/bin/mvn
      Performing Maven install for hadoop-2.6
      Using `mvn` from path: /usr/local/bin/mvn
      [ERROR] Failed to execute goal on project spark-streaming-kafka-0-8_2.12: Could not resolve dependencies for project org.apache.spark:spark-streaming-kafka-0-8_2.12:jar:spark-335572: Could not find artifact org.apache.kafka:kafka_2.12:jar:0.8.2.1 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
      

      Please note that this issue doesn't aim to update `spark-deps-hadoop-2.6` manifest here.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dongjoon Dongjoon Hyun
                Reporter:
                dongjoon Dongjoon Hyun
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: