Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8297

RocksDBListState stores whole list in single byte[]

    XMLWordPrintableJSON

    Details

      Description

      RocksDBListState currently keeps whole list of data in single RocksDB key-value pair, which implies that the list actually must fit into memory. Larger lists are not supported and end up with OOME or other error. The RocksDBListState could be modified so that individual items in list are stored in separate keys in RocksDB and can then be iterated over. A simple implementation could reuse existing RocksDBMapState, with key as index to the list and a single RocksDBValueState keeping track of how many items has already been added to the list. Because this implementation might be less efficient in come cases, it would be good to make it opt-in by a construct like

      new RocksDBStateBackend().enableLargeListsPerKey()

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              janl Jan Lukavsk√Ĺ
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m