Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-34975

FLIP-427: ForSt - Disaggregated State Store

    XMLWordPrintableJSON

Details

    Description

      This is a sub-FLIP for the disaggregated state management and its related work, please read the FLIP-423 first to know the whole story.

      As described in FLIP-423, there are some tough issues about embedded state backend on local file system, respecially when dealing with extremely large state:

      1. Constraints of local disk space complicate the prediction of storage requirements, potentially leading to job failures: Especially in cloud native deployment mode, pre-allocated local disks typically face strict capacity constraints, making it challenging to forecast the size requirements of job states. Over-provisioning disk space results in unnecessary resource overhead, while under-provisioning risks job failure due to insufficient space.
      2. The tight coupling of compute and storage resources leads to underutilization and increased waste: Jobs can generally be categorized as either CPU-intensive or IO-intensive. In a coupled architecture, CPU-intensive jobs leave a significant portion of storage resources underutilized, whereas IO-intensive jobs result in idle computing resources.

      By considering remote storage as the primary storage, all working states are maintained on the remote file system, which brings several advantages:

      1. Remote storages e.g. S3/HDFS typically offer elastic scalability, theoretically providing unlimited space.
      2. The allocation of remote storage resources can be optimized by reducing them for CPU-intensive jobs and augmenting them for IO-intensive jobs, thus enhancing overall resource utilization.
      3. This architecture facilitates a highly efficient and lightweight process for checkpointing, recovery, and rescaling through fast copy or simple move.

      This FLIP aims to realize disaggregated state for our new key-value store named ForSt which evloves from RocksDB and supports remote file system. This makes Flink get rid of the disadvantages by coupled state architecture and embrace the scalable as well as flexible cloud-native storage.

      Please see FLIP-427 for more details.

      Attachments

        Issue Links

          is a child of

          New Feature - A new feature of the product, which has yet to be developed. FLINK-34984 FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #25125

          Web Link GitHub Pull Request #25360

          Web Link GitHub Pull Request #25363

          Web Link GitHub Pull Request #25422

          1.
          		 Release beta version of ForSt Sub-task Resolved Hangxiang Yu
          2.
          		 Introduce statebackend-forst module Sub-task Resolved Hangxiang Yu
          3.
          		 Introduce ForStFileSystem to support reading and writing with ByteBuffer Sub-task Resolved Hangxiang Yu
          4.
          		 Introduce ForStStateBackend to manage ForSt Sub-task Resolved Hangxiang Yu
          5.
          		 Implement all methods of AsyncKeyedStateBakend Sub-task Resolved Hangxiang Yu
          6.
          		 Implement Map Async State API for ForStStateBackend Sub-task Resolved Yanfei Lei
          7.
          		 Implement ValueState for ForStStateBackend Sub-task Resolved Jinzhong Li
          8.
          		 Implement List Async State API for ForStStateBackend Sub-task Resolved Zakelly Lan
          9.
          		 Avoid sync waiting in coordinator thread of ForSt executor Sub-task Resolved Zakelly Lan
          10.
          		 The native metrics for column family are not reported Sub-task Open Unassigned
          11.
          		 Implement Aggregating Async State API for ForStStateBackend Sub-task Resolved Jie Pu
          12.
          		 Implement Reducing Async State API for ForStStateBackend Sub-task Resolved Yanfei Lei
          13.
          		 Make ForStFlinkFileSystem save misc file in local filesystem Sub-task Resolved Hangxiang Yu
          14.
          		 ForSt supports compiling with RocksDB Sub-task Closed Yanfei Lei
          15.
          		 Store namespace in ContextKey and state request Sub-task Resolved Yanfei Lei
          16.
          		 Support state V1 interface Sub-task Resolved Yanfei Lei
          17.
          		 Execute read/write state request in different executor Sub-task Resolved Yanfei Lei
          18.
          		 Support multiGet for forstdb Sub-task Resolved Yanfei Lei
          19.
          		 Do not reuse serialized key in Forst map state and/or other namespaces Sub-task Resolved Zakelly Lan
          20.
          		 Support dispatch state requests in task thread Sub-task Resolved Yanfei Lei
          21.
          		 Bundle forst statebackend in flink-dist and provide shortcut to enable Sub-task Resolved Zakelly Lan
          22.
          		 Graceful quit for Forst StateBackend Sub-task Resolved Zakelly Lan
          23.
          		 Optimize the overhead of writing with direct buffer in ForSt Sub-task Resolved Zakelly Lan
          24.
          		 Support file cache for ForStStateBackend Sub-task Resolved Yanfei Lei
          25.
          		 [state/forst] Refactor initialization of ForSt db core Sub-task Closed Zakelly Lan

          Activity

            People

              masteryhx Hangxiang Yu
              masteryhx Hangxiang Yu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: