[NUTCH-601] Recrawling on existing crawl directory using force option

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.0
• Patch Info: Patch Available
• Assignee: Andrzej Bialecki
• Reporter: Susam Pal
• Created: 04/Feb/08 18:08
• Updated: 10/Apr/09 12:29
• Resolved: 14/Mar/08 14:54

Description

Added a '-force' option to the 'bin/nutch crawl' command line. With this option, one can crawl and recrawl in the following manner:

bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5
bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force

This option can be used for the first crawl too:

bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force
bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force

If one tries to crawl without the -force option when the crawl directory already exists, he/she finds a small warning along with the error message:

# bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5
Exception in thread "main" java.lang.RuntimeException: crawl already
exists. Add -force option to recrawl.
       at org.apache.nutch.crawl.Crawl.main(Crawl.java:89)