[NUTCH-2148] Review and update mapred --> mapreduce config params in crawl script

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10, 2.3.1
• Fix Version/s: 1.11
• Component/s: bin
• Assignee: Lewis John McGibbney
• Reporter: Lewis John McGibbney
• Created: 21/Oct/15 01:03
• Updated: 22/Oct/15 04:54
• Resolved: 22/Oct/15 03:47

Description

Configuration parameters inside of $NUTCH_HOME/src/bin/crawl currently include

commonOptions="-D mapred.reduce.tasks=$numTasks -D mapred.child.java.opts=-Xmx1000m -D mapred.reduce.tasks.speculative.execution=false -D mapred.map.tasks.speculative.execution=false -D mapred.compress.map.output=true"

as well as

  skipRecordsOptions="-D mapred.skip.attempts.to.start.skipping=2 -D mapred.skip.map.max.skip.records=1"
  __bin_nutch parse $commonOptions $skipRecordsOptions "$CRAWL_PATH"/segments/$SEGMENT

In all honesty as part of the upgrade to Hadoop 2.4.0, this should have been addressed!!! woops.