Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
3.3.0
-
None
Description
With SPARK-39346, Spark SQL binds several exceptions to the internal error, and produces different guidance on dealing with the exception. This assumes these exceptions are only used for noticing internal bugs.
This applies to "connectors" as well, and introduces side-effect on the error log. For Kafka data source, it is a breaking and unacceptable change, because there is an important use case Kafka data source determines a case of "dataloss", and throws IllegalStateException with instruction message on workaround.
I mentioned this as "important" use case, because it can even happen with some valid scenarios - streaming query has some maintenance period and Kafka's retention on topic removes some records in the meanwhile.
Two problems arise:
1) This does not mean Spark has a bug and end users have to report, hence the guidance message on internal error is misleading.
2) Most importantly, instruction message is shown after a long stack trace. With the modification of existing test suite, I see the message being appeared in "line 90" of the error log.
We should roll the right error message back, at least for Kafka's case.
