[FLINK-34050] Rocksdb state has space amplification after rescaling with DeleteRange

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.20.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Jinzhong Li
• Reporter: Jinzhong Li
• Created: 10/Jan/24 13:35
• Updated: 28/Feb/24 11:52
• Resolved: 28/Feb/24 11:52

Description

FLINK-21321 use deleteRange to speed up rocksdb rescaling, however it will cause space amplification in some case.

We can reproduce this problem using wordCount job:

1) before rescaling, state operator in wordCount job has 2 parallelism and 4G+ full checkpoint size;

2) then restart job with 4 parallelism (for state operator),  the full checkpoint size of new job will be 8G+ ;

3) after many successful checkpoints, the full checkpoint size is still 8G+;

 

The root cause of this issue is that the deleted keyGroupRange does not overlap with current DB keyGroupRange, so new data written into rocksdb after rescaling almost never do LSM compaction with the deleted data (belonging to other keyGroupRange.)

 

And the space amplification may affect Rocksdb read performance and disk space usage after rescaling. It looks like a regression due to the introduction of deleteRange for rescaling optimization.

 

To slove this problem, I think maybe we can invoke Rocksdb.deleteFilesInRanges after deleteRange?

public static void clipDBWithKeyGroupRange() {
  //.......
  List<byte[]> ranges = new ArrayList<>();
  //.......
  deleteRange(db, columnFamilyHandles, beginKeyGroupBytes, endKeyGroupBytes);
  ranges.add(beginKeyGroupBytes);
  ranges.add(endKeyGroupBytes);
  //....

  for (ColumnFamilyHandle columnFamilyHandle : columnFamilyHandles) {
     db.deleteFilesInRanges(columnFamilyHandle, ranges, false);
  }
}

 

Issue Links

• links to: GitHub Pull Request #24367