Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4478

Implement heartbeat logic

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.2.0
    • Runtime / Coordination
    • None

    Description

      With the Flip-6 refactoring, we'll have the need for a dedicated heartbeat component. The heartbeat component is used to check the liveliness of the distributed components among each other. Furthermore, heartbeats are used to regularly transmit status updates to another component. For example, the TaskManager informs the ResourceManager with each heartbeat about the current slot allocation.

      The heartbeat is initiated from one component. This component sends a heartbeat request to another component which answers with an heartbeat response. Thus, one can differentiate between a sending and a receiving side. Apart from the triggering of the heartbeat request, the logic of treating heartbeats, marking components dead and payload delivery are the same and should be reusable by different distributed components (JM, TM, RM).

      Different models for the heartbeat reporting are conceivable. First of all, the heartbeat request could be sent as an ask operation where the heartbeat response is returned as a future on the sending side. Alternatively, the sending side could request a heartbeat response by sending a tell message. The heartbeat response is then delivered by an RPC back to the heartbeat sender. The latter model has the advantage that a heartbeat response is not tightly coupled to a heartbeat request. Such a tight coupling could cause that heartbeat response are ignored after the future has timed out even though they might still contain valuable information (receiver is still alive).

      Furthermore, different strategies for the heartbeat triggering and marking heartbeat targets as dead are conceivable. For example, we could periodically (with a fixed period) trigger a heartbeat request and mark all targets as dead if we didn't receive a heartbeat response in a given time period. Furthermore, we could adapt the heartbeat interval and heartbeat timeouts with respect to the latency of previous heartbeat responses. This would reflect the current load and network conditions better.

      For the first version, I would propose to use a fixed period heartbeat with a maximum heartbeat timeout before a target is marked dead. Furthermore, I would propose to use tell messages (fire and forget) to request and report heartbeats because they are the more flexible model imho.

      Attachments

        Issue Links

          Blocked

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-6120 Implement heartbeat logic between JobManager and ResourceManager

          • Major - Major loss of function.
          • Resolved
          blocks

          Sub-task - The sub-task of the issue FLINK-4449 Heartbeat Manager integration with TaskExecutor

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue FLINK-4551 Heartbeat Manager integration with JobMaster

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue FLINK-4354 Implement heartbeat logic between TaskManager and ResourceManager

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-4364 Implement heartbeat logic between TaskManager and JobManager

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-4319 Rework Cluster Management (FLIP-6)

          • Major - Major loss of function.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue FLINK-4354 Implement heartbeat logic between TaskManager and ResourceManager

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-4364 Implement heartbeat logic between TaskManager and JobManager

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-6120 Implement heartbeat logic between JobManager and ResourceManager

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Design document for the HeartbeatManager

          Web Link GitHub Pull Request #2435

          There are no Sub-Tasks for this issue.

          Activity

            People

              trohrmann Till Rohrmann
              trohrmann Till Rohrmann
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: