Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-26759

Legacy source support waiting for recordWriter to be available

    XMLWordPrintableJSON

Details

    Description

      In order for Unaligned Checkpoint not to be blocked, StreamTask#processInput will check recordWriter.isAvailable(). If not available, the data will not be processed until recordWriter is available.

      The new Source api is compatible with the above logic, but Legacy Source is not compatible with the above logic. When using Unaligned Checkpoint, if the backpressure of Legacy Source is high, the Checkpoint duration of Legacy Source will be very long.

       

      Since legacy sources are often used in production, can we add logic to wait for recordWriter to be available for legacy source?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              fanrui Rui Fan
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: