[NUTCH-1219] Upgrade all jobs to new MapReduce API

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.15
• Assignee: Unassigned
• Reporter: Markus Jelsma
• Created: 13/Dec/11 16:46
• Updated: 01/Oct/19 14:29
• Resolved: 11/Apr/18 10:59

Description

We should upgrade to the new Hadoop API for Nutch trunk as already has been done for the Nutchgora branch. If i'm not mistaken we can already upgrade to the latest 0.20.5 version that still carries the legacy API so we can, without immediately upgrading to 0.21 or higher, port the jobs to the new API without having the need for a separate branch to work on.

To the committers who created/ported jobs in NutchGora, please write down your advice and experience.

http://www.slideshare.net/sh1mmer/upgrading-to-the-new-map-reduce-api

Issue Links

• is superceded by: NUTCH-2375 Upgrade the code base from org.apache.hadoop.mapred to org.apache.hadoop.mapreduce

Sub-Tasks

• Migrate DomainStatistics to MapReduce API: Closed
• Migrate WebGraph to MapReduce API: Closed
• Migrate FreeGenerator to MapReduce API: Closed
• Migrate CrawlDBScanner to MapReduce API: Closed
• Migrate CrawlDbReader to MapReduce API: Closed