[HDFS-12213] Ozone: Corona: Support for online mode

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: ozone
• Labels: OzonePostMerge tool
• Assignee: Nandakumar
• Reporter: Nandakumar
• Created: 28/Jul/17 10:30
• Updated: 07/Jul/22 06:05
• Resolved: 07/Jul/22 06:05

Description

This jira brings support for online mode in corona.
In online mode, common crawl data from AWS will be used to populate ozone with data. Default source is CC-MAIN-2017-17/warc.paths.gz (it contains the path to actual data segment), user can override this using -source.
The following values are derived from URL of Common Crawl data

• Domain will be used as Volume
• URL will be used as Bucket
• FileName will be used as Key