[NUTCH-1949] Dump out the Nutch data into the Common Crawl format

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10
• Component/s: crawldb, linkdb, storage, tool
• Assignee: Lewis John McGibbney
• Reporter: Giuseppe Totaro
• Created: 24/Feb/15 04:40
• Updated: 04/Mar/15 21:31
• Resolved: 04/Mar/15 18:49

Description

We are going to develop a CommonCrawlDataDumper.java class. The CommonCrawlDataDumper is a tool able to perfom the following steps:

1. deserialize the crawled data from Nutch
2. map serialized data on the proper JSON structure
3. serialize the data into CBOR format
4. optionally, compress the serialized data using gzip

This tool has to be able to work with either single Nutch segments or directory including segments as input data.

Thanks Lewis John McGibbney and Chris A. Mattmann for your great suggestions, support and code.

Sub-Tasks

• CommonCrawlDumper : Invalid format + skipped parts: Resolved
• Make CommonCrawlFormatJackson instance reusable by properly handling object state: Resolved