[FLINK-35738] Release Testing: Verify FLINK-26050 Too many small sst files in rocksdb state backend when using time window created in ascending order

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Runtime / State Backends
• Assignee: Samrat Deb
• Reporter: Rui Fan
• Created: 4 days ago 06:58
• Updated: 4 days ago 07:47

Description

The problem occurs when using RocksDB and specific queries/jobs (please see the ticket for the detailed description).

To test the solution, run the following query with RocksDB as a state backend:

 

INSERT INTO top_5_highest_view_time
SELECT *
FROM   (
                SELECT   *,
                         ROW_NUMBER() OVER (PARTITION BY window_start, window_end ORDER BY view_time DESC) AS rownum
                FROM     (
                                  SELECT   window_start,
                                           window_end,
                                           product_id,
                                           SUM(view_time) AS view_time,
                                           COUNT(*)       AS cnt
                                  FROM     TABLE(TUMBLE(TABLE `shoe_clickstream`, DESCRIPTOR($rowtime), INTERVAL '10' MINUTES))
                                  GROUP BY window_start,
                                           window_end,
                                           product_id))
WHERE  rownum <= 5;

 

With the feature disabled (default), the number of files in rocksdb working directory (as well as in the checkpoint) should grow indefinitely.

 

With feature enabled, the number of files should stays constant (as they should get merged with each other).

To enable the feature, set 

state.backend.rocksdb.manual-compaction.min-interval

 set to 1 minute for example.

 

Please consult https://github.com/apache/flink/blob/e7d7db3b6f87e53d9bace2a16cf95e5f7a79087a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/sstmerge/RocksDBManualCompactionOptions.java#L29 for other options if necessary.