Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Currently when an operator invokes kill(), only the upstream operators within that fragment get terminated. This causes problems if the upstream operator in a different fragment is still continuously producing batches, and it then becomes the responsibility of the operator to drain all the batches even though a kill was issued.
While dealing with plans with multiple fragments we would need to inform the sender that a kill has been invoked. It would then be the responsibility of the sender to not push any more batches to the corresponding receiver.