Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-23527

Clarify `SourceFunction#cancel()` contract about interrupting

    XMLWordPrintableJSON

Details

    • Hide
      Contract of the SourceFunction#cancel() method with respect to interruptions has been clarified:
      - source itself shouldn’t be interrupting the source thread
      - interrupt shouldn’t be expected in the clean cancellation case
      Show
      Contract of the SourceFunction#cancel() method with respect to interruptions has been clarified: - source itself shouldn’t be interrupting the source thread - interrupt shouldn’t be expected in the clean cancellation case

    Description

      We should clarify the contract of SourceFunction#cancel()

      1. source itself shouldn’t be interrupting the source thread
      2. interrupt shouldn’t be expected in the clean cancellation case

      Interrupting the code on the clean shutdown path can cause failures when doing `stop-with-savepoint`. If source thread is interrupted during backpressure, this leaves network stack in invalid state, making it impossible to send EndOfPartitionEvent to complete the shutdown.

      In a bit more detail, when source thread is backpressured, network stack might have already sent a partial record and it could be waiting for a buffer to finish writing/serialising that record. If network stack is interrupted while waiting for that buffer, it will never resume writing/serialisation of the remaining part of that record, while downstream node will be expecting those bytes. If in this situation we attempt to emit anything (like EndOfPartitionEvent), this will most likely cause deserialisation errors on the downstream nodes.

      Attachments

        1. master_preferences
          0.0 kB
          vickyshiv

        Issue Links

          Activity

            People

              sewen Stephan Ewen
              pnowojski Piotr Nowojski
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: