[FLINK-29402] Add USE_DIRECT_READ configuration parameter for RocksDB

• Type: Improvement
• Status: Closed
• Priority: Not a Priority
• Resolution: Not A Problem
• Affects Version/s: 1.16.0
• Fix Version/s: 1.17.0
• Component/s: Runtime / State Backends
• Labels: Enhancement pull-request-available rocksdb
• Release Note: The changes do not break previous releases.
• Flags: Patch
• Assignee: Unassigned
• Reporter: Donatien
• Created: 23/Sep/22 13:56
• Updated: 27/Nov/22 02:44
• Resolved: 20/Oct/22 09:05

Description

RocksDB allows the use of DirectIO for read operations to bypass the Linux Page Cache. To understand the impact of Linux Page Cache on performance, one can run a heavy workload on a single-tasked Task Manager with a container memory limit identical to the TM process memory. Running this same workload on a TM with no container memory limit will result in better performances but with the host memory exceeding the TM requirement.

Linux Page Cache are of course useful but can give false results when benchmarking the Managed Memory used by RocksDB. DirectIO is typically enabled for benchmarks on working set estimation [Zwaenepoel et al.|https://arxiv.org/abs/1702.04323.]

I propose to add a configuration key allowing users to enable the use of DirectIO for reads thanks to the RocksDB API. This configuration would be disabled by default.

Issue Links

• links to: GitHub Pull Request #20903