Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13698

Rework threading model of CheckpointCoordinator

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Currently CheckpointCoordinator and CheckpointFailureManager code is executed by multiple different threads (mostly ioExecutor, but not only). It's causing multiple concurrency issues, for example: https://issues.apache.org/jira/browse/FLINK-13497

      Proper fix would be to rethink threading model there. At first glance it doesn't seem that this code should be multi threaded, except of parts doing the actual IO operations, so it should be possible to run everything in one single ExecutionGraph's thread and just run asynchronously necessary IO operations with some feedback loop ("mailbox style").

      I would strongly recommend fixing this issue before adding new features in the CheckpointCoordinator component.

      Attachments

        Issue Links

        causes

        Bug - A problem which impairs or prevents the functions of the product. FLINK-13497 Checkpoints can complete after CheckpointFailureManager fails job

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed
        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-5960 Make CheckpointCoordinator less blocking

        • Major - Major loss of function.
        • Closed
        is related to

        Bug - A problem which impairs or prevents the functions of the product. FLINK-15132 Checkpoint Coordinator does Checkpoint I/O in JobMaster Main Thread

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed
        relates to

        Bug - A problem which impairs or prevents the functions of the product. FLINK-19401 Job stuck in restart loop due to excessive checkpoint recoveries which block the JobMaster

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. FLINK-26306 [Changelog] Thundering herd problem with materialization

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-26590 Triggered checkpoints can be delayed by discarding shared state

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-16931 Large _metadata file lead to JobManager not responding when restart

        • Not a Priority -
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-5960 Make CheckpointCoordinator less blocking

        • Major - Major loss of function.
        • Closed
        links to

        Web Link Refactor Thread Model of CheckpointCoordinator

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            pnowojski Piotr Nowojski
            Votes:
            1 Vote for this issue
            Watchers:
            19 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 20m
                1h 20m

                Slack

                  Issue deployment