Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-23527

Clarify `SourceFunction#cancel()` contract about interrupting

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      Contract of the SourceFunction#cancel() method with respect to interruptions has been clarified:
      - source itself shouldn’t be interrupting the source thread
      - interrupt shouldn’t be expected in the clean cancellation case
      Show
      Contract of the SourceFunction#cancel() method with respect to interruptions has been clarified: - source itself shouldn’t be interrupting the source thread - interrupt shouldn’t be expected in the clean cancellation case

      Description

      We should clarify the contract of SourceFunction#cancel()

      1. source itself shouldn’t be interrupting the source thread
      2. interrupt shouldn’t be expected in the clean cancellation case

      Interrupting the code on the clean shutdown path can cause failures when doing `stop-with-savepoint`. If source thread is interrupted during backpressure, this leaves network stack in invalid state, making it impossible to send EndOfPartitionEvent to complete the shutdown.

      In a bit more detail, when source thread is backpressured, network stack might have already sent a partial record and it could be waiting for a buffer to finish writing/serialising that record. If network stack is interrupted while waiting for that buffer, it will never resume writing/serialisation of the remaining part of that record, while downstream node will be expecting those bytes. If in this situation we attempt to emit anything (like EndOfPartitionEvent), this will most likely cause deserialisation errors on the downstream nodes.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sewen Stephan Ewen
                Reporter:
                pnowojski Piotr Nowojski
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: