[SPARK-27550] Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12 - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 2.4.3
Fix Version/s: 2.4.3
Component/s: Tests
Labels:
None

Description

Since ~~SPARK-27274~~ deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more.

Kakfa 0.8 doesn't have Scala-2.12 artifacts, e.g., `org.apache.kafka:kafka_2.12:jar:0.8.2.1`. This issue aims to fix `test-dependencies.sh` script to understand Scala binary version.

$ dev/change-scala-version.sh 2.12
$ dev/test-dependencies.sh
Using `mvn` from path: /usr/local/bin/mvn
Using `mvn` from path: /usr/local/bin/mvn
Performing Maven install for hadoop-2.6
Using `mvn` from path: /usr/local/bin/mvn
[ERROR] Failed to execute goal on project spark-streaming-kafka-0-8_2.12: Could not resolve dependencies for project org.apache.spark:spark-streaming-kafka-0-8_2.12:jar:spark-335572: Could not find artifact org.apache.kafka:kafka_2.12:jar:0.8.2.1 in central (https://repo.maven.apache.org/maven2) -> [Help 1]

Please note that this issue doesn't aim to update `spark-deps-hadoop-2.6` manifest here.