Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-32833

Rocksdb CacheIndexAndFilterBlocks must be true when using shared memory

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.17.1
    • None
    • None

    Description

      Currently in RocksDBResourceContainer#getColumnOptions, if sharedResources is used, blockBasedTableConfig will add the following configuration by default.

      blockBasedTableConfig.setBlockCache(blockCache);
      blockBasedTableConfig.setCacheIndexAndFilterBlocks(true);
      blockBasedTableConfig.setCacheIndexAndFilterBlocksWithHighPriority(true);
      blockBasedTableConfig.setPinL0FilterAndIndexBlocksInCache(true);

      In my understanding, these configurations can help flink better manage the memory of rocksdb and save some memory overhead in some scenarios. But this may not be the best practice, mainly for the following reasons:
      1. After CacheIndexAndFilterBlocks is set to true, it may cause index and filter miss when reading, resulting in performance degradation.
      2. These parameters may not be bound together with whether shared memory is used, or some configurations should be supported separately to decide whether to enable these features

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mayuehappy Yue Ma
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: