Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13698

Rework threading model of CheckpointCoordinator

    XMLWordPrintableJSON

Details

    Description

      Currently CheckpointCoordinator and CheckpointFailureManager code is executed by multiple different threads (mostly ioExecutor, but not only). It's causing multiple concurrency issues, for example: https://issues.apache.org/jira/browse/FLINK-13497

      Proper fix would be to rethink threading model there. At first glance it doesn't seem that this code should be multi threaded, except of parts doing the actual IO operations, so it should be possible to run everything in one single ExecutionGraph's thread and just run asynchronously necessary IO operations with some feedback loop ("mailbox style").

      I would strongly recommend fixing this issue before adding new features in the CheckpointCoordinator component.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. FLINK-13497 Checkpoints can complete after CheckpointFailureManager fails job

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-5960 Make CheckpointCoordinator less blocking

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-15132 Checkpoint Coordinator does Checkpoint I/O in JobMaster Main Thread

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-19401 Job stuck in restart loop due to excessive checkpoint recoveries which block the JobMaster

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. FLINK-26306 [Changelog] Thundering herd problem with materialization

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-26590 Triggered checkpoints can be delayed by discarding shared state

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-16931 Large _metadata file lead to JobManager not responding when restart

          • Not a Priority -
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-5960 Make CheckpointCoordinator less blocking

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Refactor Thread Model of CheckpointCoordinator

          Activity

            People

              Unassigned Unassigned
              pnowojski Piotr Nowojski
              Votes:
              1 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m