Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11105 Add a new implementation of the HighAvailabilityServices using etcd
  3. FLINK-11811

Implement etcd based RunningJobsRegistry and SubmittedJobGraphStore

Agile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Minor
    • Resolution: Abandoned
    • None
    • None
    • Runtime / Coordination
    • None

    Description

      Zookeeper based RunningJobsRegistry uses zookeeper as a key-value store to keep the latest job statuses using persistent zookeeper nodes. So etcd based RunningJobsRegistry can use io.etcd.jetcd.KV to achieve the same goal.

      Zookeeper based SubmittedJobGraphStore applies ZooKeeperStateHandleStore to store submitted each job graph in a dfs file while records its metadata, such as dfs location, to a zookeeper node in a zookeeper directory. etcd based SubmittedJobGraphStore will use EtcdSubmittedJobGraphStore to fulfill the target.

      In addition, SubmittedJobGraphStore needs to watch this zookeeper directory to get notifications once znodes in such directory is created or deleted. Previous etcd API io.etcd.jetcd.Watch can watch a key as prefix by setting io.etcd.jetcd.options.WatchOption#withPrefix() which is just like watching a directory in zookeeper.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            ssy MalcolmSanders
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment