[NUTCH-2095] WARC exporter for the CommonCrawlDataDumper

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Affects Version/s: 1.11
• Component/s: commoncrawl, tool
• Labels: tools warc
• Patch Info: Patch Available
• Flags: Patch
• External issue URL: https://github.com/apache/nutch/pull/55
• Assignee: Unassigned
• Reporter: Jorge Luis Betancourt Gonzalez
• Created: 11/Sep/15 19:17
• Updated: 28/Jan/21 14:04
• Resolved: 23/Jul/18 11:50

Description

Adds the possibility of exporting the nutch segments to a WARC files.

From the usage point of view a couple of new command line options are available:

-warc: enables the functionality to export into WARC files, if not specified the default JACKSON formatter is used.
-warcSize: enable the option to define a max file size for each WARC file, if not specified a default of 1GB per file is used as recommended by the WARC ISO standard.

The usual -gzip flag can be used to enable compression on the WARC files.

Some changes to the default CommonCrawlDataDumper were done, essentially some changes to the Factory and to the Formats. This changes avoid creating a new instance of a CommmonCrawlFormat on each URL read from the segments.