Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18578

Add rejecting checkpoint logic in source

    XMLWordPrintableJSON

    Details

      Description

      Under some database's change data capture (CDC) case, the process is usually divided into two phases: snapshotting phase (lock captured tables and scan all records in them) and log streaming phase (read all changes starting from the moment of locking tables) in order to build a complete view of captured tables. The first snapshotting phase should be atomic so we have to give up all records created in snapshotting phase if any failure happen, because contents in captured tables might have changed during recovery. And checkpointing within snapshotting phase is meaningless too.

      As a result, we need to add a new feature in the source to reject checkpoint if the source is currently within an atomic operation or some other processes that cannot do a checkpoint currently. This rejection should not be treated as a failure that could lead to failure of the entire job. 
       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              renqs Qingsheng Ren
            • Votes:
              2 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated: