Details
-
Improvement
-
Status: Reopened
-
Not a Priority
-
Resolution: Unresolved
-
None
-
None
-
None
Description
There's no way to retrieve more information about corrupted message in the DeserializationSchema class. It is only possible to return null, which is a signal to skip the message, and to throw an exception, which will cause job failure.
For investigation purposes it would be good to have more information, like:
- kafka topic from which the message came from
- in Flink 1.4, it is possible to subscribe using Pattern, so topic name is not always evident
- kafka topic offset
The idea is to write this information into the log file for further analysis. Having topic name and offset allows to retrieve the message and investigate it.