A user reported this issue on the firstname.lastname@example.org mailing list and analyzed the situation.
- A program that reads from Kafka and computes counts in a keyed 15 minute tumbling window. StateBackend is RocksDB and checkpointing is enabled.
.reduce(new ReduceFunction(), new WindowFunction())
- At some point HDFS went into a safe mode due to NameNode issues
- The following exception was thrown
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category WRITE is not supported in state standby. Visit https: ..................
- The pipeline came back after a few restarts and checkpoint failures, after the HDFS issues were resolved.
- It was evident that operator state was lost. Either it was the Kafka consumer that kept on advancing it's offset between a start and the next checkpoint failure (a minute's worth) or the the operator that had partial aggregates was lost.
The user did some in-depth analysis (see mail thread) and might have (according to Aljoscha Krettek) identified the problem.
Stefan Richter, can you have a look at this issue and check if it is relevant?