As noted in
SPARK-27550 we want to encourage testing of Spark 2.4.x with Scala-2.12, and kafka-0-8 does not support Scala-2.12.
Currently, the PySpark tests invoked by `python/run-tests` demand the presence of kafka-0-8 libraries. If not present, this failure message will be generated:
This change is only targeted at branch-2.4, as most kafka-0-8 related materials have been removed in master and this problem no longer occurs there.
The proposed solution is to make the kafka-0-8 stream testing optional for pyspark testing, exactly the same as the Kinesis stream testing currently is, in file `python/pyspark/streaming/tests.py`. This is only a few lines of change.
Ideally it would be limited to when SPARK_SCALA_VERSION >= 2.12, but it turns out to be somewhat onerous to reliably obtain that value from within the python test env, and no other python test code currently does so. So my proposed solution simply makes the use of the kafka-0-8 profile optional, and leaves it to the tester to include it for Scala-2.11 test builds and exclude it for Scala-2.12 test builds.
PR will be available in a day or so.