[FLINK-35573] [FLIP-447] Upgrade FRocksDB from 6.20.3 to 8.10.0

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Runtime / State Backends
• Assignee: Yue Ma
• Reporter: Yue Ma
• Created: 12/Jun/24 08:56
• Updated: 12/Oct/24 02:15

Description

The FLIP: https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0 

RocksDBStateBackend is widely used by Flink users in large state scenarios.The last upgrade of FRocksDB was in version Flink-1.14, which mainly supported features such as support arm platform, deleteRange API, period compaction, etc. It has been a long time since then, and RocksDB has now been released to version 8.x. The main motivation for this upgrade is to leverage the features of higher versions of Rocksdb to make Flink RocksDBStateBackend more powerful. While RocksDB is also continuously optimizing and bug fixing, we hope to keep FRocksDB more or less in sync with RocksDB and upgrade it periodically.

The plan is as follows

Release Frocksdb-8.10.0 official products

1. Prepare the compiled frocksdb branch
2. Support Disable PERF-CONTEXT in compilation
3. Cherry pick IngestDB requires  commit
4. Setup the CI environment for FRocksDB-8.10
5. Release Frocksdb-8.10.0 official products
6. Update the dependency of FRocksDB in pom file of Flink-RocksDB-Statebackend

Make ingestDB available 

1. Flink ingestDB code related bug fixes
2. Remove comments from the code related to ingestDB and mark the functionality of ingestDB as available (perhaps we can mark ingestDB as an experimental feature first)
3. Using ingestDB as the default recovery mode for rescaling

（已编辑）

Issue Links

• is duplicated by: FLINK-33338 Bump FRocksDB version
• mentioned in: Page Loading...

Sub-Tasks

• Setup base branch for FrocksDB-8.10: Closed
• FRocksDB supports disabling perf context during compilation: Closed
• FRocksDB cherry pick IngestDB related commits: Closed
• Setup the CI environment for FRocksDB-8.10: Resolved
• Release Frocksdb-8.10.0 official products: Open
• Update the FrocksDB version in FLINK: Resolved
• Improvements and fix ingestDB recovery mode related bugs: Closed
• Remove comments from the code related to ingestDB: Closed
• Marking ingestDB as the default recovery mode for rescaling: Open