[SPARK-45029] Ignore `from_protobuf messageClassName` and `from_protobuf messageClassName options` in `PlanGenerationTestSuite`

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0, 4.0.0
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Connect, Tests
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 31/Aug/23 07:43
• Updated: 01/Sep/23 02:53
• Resolved: 01/Sep/23 02:53

Description

The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` module when testing the connect module. This allows `mvn clean install` and `mvn package test` to successfully pass tests. But if `mvn clean test` is executed directly, an error in `package org.sparkproject.spark_protobuf.protobuf does not exist` will occur. This is because `mvn clean test` directly uses the classes file of the `spark-protobuf` module for testing, without the 'package', hence it does not `shade` and `relocate` protobuf. 

On the other hand, SPARK-43646 breaks the usability of importing Spark as a Maven project into IDEA. 

Therefore, the changes of SPARK-43646 be reverted in https://github.com/apache/spark/pull/42746. 

In summary, when using Maven to test the connect module:

• sometimes the shaded spark-protobuf jar will be used, such as 'mvn package test',
• sometimes the unshaded classes directory will be used, such as 'mvn clean test'.

Therefore, there is no perfect solution to pass both tests, so 'from_protobuf messageClassName' and 'from_protobuf messageClassName' in 'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled when we find a better solution.

Issue Links

• is related to: SPARK-45030 Re-enable `from_protobuf messageClassName` and `from_protobuf messageClassName options` in `PlanGenerationTestSuite`