Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-23527

Clarify `SourceFunction#cancel()` contract about interrupting

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      Contract of the SourceFunction#cancel() method with respect to interruptions has been clarified:
      - source itself shouldn’t be interrupting the source thread
      - interrupt shouldn’t be expected in the clean cancellation case
      Show
      Contract of the SourceFunction#cancel() method with respect to interruptions has been clarified: - source itself shouldn’t be interrupting the source thread - interrupt shouldn’t be expected in the clean cancellation case

      Description

      We should clarify the contract of SourceFunction#cancel()

      1. source itself shouldn’t be interrupting the source thread
      2. interrupt shouldn’t be expected in the clean cancellation case

      Interrupting the code on the clean shutdown path can cause failures when doing `stop-with-savepoint`. If source thread is interrupted during backpressure, this leaves network stack in invalid state, making it impossible to send EndOfPartitionEvent to complete the shutdown.

      In a bit more detail, when source thread is backpressured, network stack might have already sent a partial record and it could be waiting for a buffer to finish writing/serialising that record. If network stack is interrupted while waiting for that buffer, it will never resume writing/serialisation of the remaining part of that record, while downstream node will be expecting those bytes. If in this situation we attempt to emit anything (like EndOfPartitionEvent), this will most likely cause deserialisation errors on the downstream nodes.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              sewen Stephan Ewen
              Reporter:
              pnowojski Piotr Nowojski

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment