A user reported this issue on the email@example.com mailing list and analyzed the situation.
- A program that reads from Kafka and computes counts in a keyed 15 minute tumbling window. StateBackend is RocksDB and checkpointing is enabled.
- At some point HDFS went into a safe mode due to NameNode issues
- The following exception was thrown
- The pipeline came back after a few restarts and checkpoint failures, after the HDFS issues were resolved.
- It was evident that operator state was lost. Either it was the Kafka consumer that kept on advancing it's offset between a start and the next checkpoint failure (a minute's worth) or the the operator that had partial aggregates was lost.
Stefan Richter, can you have a look at this issue and check if it is relevant?