[FLINK-35501] Use common thread pools when transferring RocksDB state files

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.20.0
• Fix Version/s: 1.20.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Release Note: The semantics of state.backend.rocksdb.checkpoint.transfer.thread.num changed slightly: If negative, the common (TM) IO thread pool is used (see cluster.io-pool.size) for up/downloading RocksDB files
• Assignee: Roman Khachatryan
• Reporter: Roman Khachatryan
• Created: 31/May/24 18:23
• Updated: 06/Jun/24 12:21
• Resolved: 06/Jun/24 12:21

Description

Currently, each RocksDB state backend creates an executor backed by a thread pool.

This makes it difficult to control the total number of threads per TM because it might have at least one task per slot and theoretically, many state backends per task (because of chaining).

Additionally, using a common thread pool allows to indirectly control the load on the underlying DFS (e.g. the total number of requests to S3 from a TM).

Issue Links

• links to: GitHub Pull Request #24882