Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10471

State TTL cleanup using RocksDb compaction filter

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      The used build of FRocksdb is based on the upgraded version 5.17.2 of Rocksdb. In case of Mac OS X, the Rocksdb version 5.17.2 is supported only for OS X version >= 10.13.
      See also: https://github.com/facebook/rocksdb/issues/4862
      Show
      The used build of FRocksdb is based on the upgraded version 5.17.2 of Rocksdb. In case of Mac OS X, the Rocksdb version 5.17.2 is supported only for OS X version >= 10.13. See also: https://github.com/facebook/rocksdb/issues/4862

      Description

      In case of rocksdb backend, we can piggy back compaction using custom compaction filter which checks our last access timestamp and discards expired values. It requires contributing a C++ Flink TTL specific filter to Rocksdb, like for cassandra. At the moment RocksDB does not support compaction filter plugins (see PR discussion), it is under development.

      Meanwhile, we can apply to strategies to enable this feature in Flink:

      • Release and maintain a temporary fork of RocksDB for Flink: FRocksDB and merge TTL filter into this fork (used in Flink 1.8)
      • Build C++ TTL filter separately, pack this C++ lib into its JNI java client jar and load it in Flink additionally to vanila RocksDB (Flink RocksDB extensions, planned for Flink 1.9)

      The second strategy is more flexible in the long run.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                azagrebin Andrey Zagrebin
                Reporter:
                azagrebin Andrey Zagrebin
              • Votes:
                1 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m