Details
-
Improvement
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
None
-
None
Description
Motivated by this email thread: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/KafkaFetcher-closed-before-end-of-stream-is-received-for-all-partitions-td32827.html
In general, the isEndOfStream method is ill-defined. An possible improvement is to redefine the semantics and method signature, to:
boolean isEndOfPartition(T nextElement, String topic, int partition);
so that it indicates end of a Kafka partition. With this, the Kafka consumer is able to properly define end-of-subtask to be when all of its assigned partitions have signaled EOF.