[SPARK-24279] Incompatible byte code errors, when using test-jar of spark sql.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.3.0, 2.4.0
• Component/s: SQL, Structured Streaming, Tests
• Assignee: Unassigned
• Reporter: Prashant Sharma
• Created: 15/May/18 10:47
• Updated: 07/Jun/18 05:04
• Resolved: 07/Jun/18 05:04

Description

Using test libraries available with spark sql streaming, produces weird incompatible byte code errors. It is already tested on a different virtual box instance to make sure, that it is not related to my system environment. A reproducer is uploaded to github.

https://github.com/ScrapCodes/spark-bug-reproducer

 

Doing a clean build reproduces the error.

 

Verbatim paste of the error.

[INFO] Compiling 1 source files to /home/prashant/work/test/target/test-classes at 1526380360990
[ERROR] error: missing or invalid dependency detected while loading class file 'QueryTest.class'.
[INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[INFO] A full rebuild may help if 'QueryTest.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtilsBase.class'.
[INFO] Could not access type PlanTestBase in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] error: missing or invalid dependency detected while loading class file 'SQLTestUtils.class'.
[INFO] Could not access type PlanTest in package org.apache.spark.sql.catalyst.plans,
[INFO] because it (or its dependencies) are missing. Check your build definition for
[INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against an incompatible version of org.apache.spark.sql.catalyst.plans.
[ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25: error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
[ERROR] val inputData = MemoryStream[Int]
[ERROR] ^
[ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30: error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
[ERROR] CheckAnswer(2, 3, 4))
[ERROR] ^
[ERROR] 5 errors found