[SPARK-43067] Error class resource file in Kafka connector is misplaced

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.1, 3.5.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 07/Apr/23 10:17
• Updated: 08/Apr/23 22:19
• Resolved: 08/Apr/23 21:39

Description

SPARK-41387 adopted error class framework in Kafka connector. Since it's a connector, the error class file is intentionally put to the Kafka connector module instead of being added to the Spark's central error class file.

Unfortunately I've figured out somehow that the place is wrong. It should be placed to the resources directory in source, not test. The problem cannot be captured in test suite as it's available in test artifact, but I can trigger the problem via adding Kafka connector jar into classpath and initialize KafkaExceptions object.

Hopefully the blast radius of the problem is trivial as Kafka connector uses error class only for assertions which should not be triggered unless some accident happens e.g. topic is deleted and recreated while the streaming query is running with Trigger.Available.