[NUTCH-478] Add function for stopping FetherThread gracefully

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9.0
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: chee.wu
• Created: 05/May/07 06:27
• Updated: 13/Apr/11 22:58
• Resolved: 13/Apr/11 22:58

Description

Now the fetch process will be stopped only when time out occurred during the fetch:
"System.currentTimeMillis() - lastRequestStart.get()) > timeout "

We don't have method to let fetch process to stop.Some times we may have strict time requirement for fetch process, for example from 11pm to 7am.I want to shutdown fetch process at 7am every day even there still have pages remained unfeched in the segments generated.

A possible solution to implement this might be:
1. User create a file named "FetchStop" in nutch home.
2. Check the existence of the file every minute in the main thread,and set the boolean variable like "stopFetch" to true;
3. FetchThread will check the status of "stopFetch" before fetching next URL. If changed to true, FetcherThread will stop right now,also the value of activeThreads will be reduced.
4. Finally, the main thread will end if activeThreads=0