[HDDS-4395] Ozone Data Generator for Fast Scale Test

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Tools
• Labels: pull-request-available
• Assignee: Marton Elek
• Reporter: Wei-Chiu Chuang
• Created: 26/Oct/20 16:37
• Updated: 17/Mar/21 08:30
• Resolved: 17/Mar/21 08:30

Description

I've been working on this fun project and would like to share with the community.

 

Synopsis

We want to prove Ozone runs well at scale, in terms of number of keys (billions of keys), as well as dense DataNodes where each DN has hundreds of TB or even PB-scale capacity.

Challenge: Data generation

The challenge is to generate a huge data set fast so that we can benchmark the system quickly. No existing tool is capable at this scale.

 

Proposal:

The major bottleneck is OM’s key insertion performance. In addition, Ozone uses a single pipeline to write data, unless multi-raft is enabled.

 

Instead of using Ozone's client API to generate data, We should write directly to OM, SCM and DN’s rocksdb. RocksDB can support up to a million key bulk load operations.

 

Similarly, we can skip the normal Ozone client write path; populate the container db and block files directly.

 

(more details in the design doc)

Issue Links

• links to: GitHub Pull Request #1826