[FLINK-3718] Add Option For Completely Async Backup in RocksDB State Backend

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Assignee: Aljoscha Krettek
• Reporter: Aljoscha Krettek
• Created: 08/Apr/16 11:42
• Updated: 28/Feb/19 11:14
• Resolved: 20/Apr/16 14:17

Description

Right now, the snapshotting for RocksDB has a synchronous part where a backup of the RocksDB database is drawn and an asynchronous part where this backup is written to HDFS.

We should add an option that uses the snapshot feature of RocksDB to get an iterator over all keys at a set point in time. The iterator can be used to store everything to HDFS. Normal operation can continue while we store the keys. This makes the snapshot completely asynchronous.