[HDFS-12071] Ozone: Corona: Implementation of Corona

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDFS-7240
• Component/s: ozone
• Labels: tool
• Hadoop Flags: Reviewed
• Assignee: Nandakumar
• Reporter: Nandakumar
• Created: 29/Jun/17 22:03
• Updated: 26/Apr/18 21:52
• Resolved: 21/Jul/17 01:07

Description

Tool to populate ozone with data for testing.
This is not a map-reduce program and this is not for benchmarking Ozone write throughput.
It supports both online and offline modes. Default mode is offline, -mode can be used to change the mode.

In online mode, active internet connection is required, common crawl data from AWS will be used. Default source is CC-MAIN-2017-17/warc.paths.gz (it contains the path to actual data segment), user can override this using -source.
The following values are derived from URL of Common Crawl data

• Domain will be used as Volume
• URL will be used as Bucket
• FileName will be used as Key

In offline mode, the data will be random bytes and size of data will be 10 KB.

• Default number of Volumes 10, -numOfVolumes can be used to override
• Default number of Buckets per Volume 1000, -numOfBuckets can be used to override
• Default number of Keys per Bucket 500000, -numOfKeys can be used to override