Details
Description
The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` module when testing the connect module. This allows `mvn clean install` and `mvn package test` to successfully pass tests. But if `mvn clean test` is executed directly, an error in `package org.sparkproject.spark_protobuf.protobuf does not exist` will occur. This is because `mvn clean test` directly uses the classes file of the `spark-protobuf` module for testing, without the 'package', hence it does not `shade` and `relocate` protobuf.
On the other hand, SPARK-43646 breaks the usability of importing Spark as a Maven project into IDEA.
Therefore, the changes of SPARK-43646 be reverted in https://github.com/apache/spark/pull/42746.
In summary, when using Maven to test the connect module:
- sometimes the shaded spark-protobuf jar will be used, such as 'mvn package test',
- sometimes the unshaded classes directory will be used, such as 'mvn clean test'.
Therefore, there is no perfect solution to pass both tests, so 'from_protobuf messageClassName' and 'from_protobuf messageClassName' in 'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled when we find a better solution.
Attachments
Attachments
Issue Links
- is related to
-
SPARK-45030 Re-enable `from_protobuf messageClassName` and `from_protobuf messageClassName options` in `PlanGenerationTestSuite`
- Open