[NUTCH-61] Adaptive re-fetch interval. Detecting umodified content

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: fetcher
• Assignee: Andrzej Bialecki
• Reporter: Andrzej Bialecki
• Due: 20/Jun/05
• Created: 06/Jun/05 05:38
• Updated: 10/Apr/09 12:29
• Resolved: 30/May/07 18:35

Description

Currently Nutch doesn't adjust automatically its re-fetch period, no matter if individual pages change seldom or frequently. The goal of these changes is to extend the current codebase to support various possible adjustments to re-fetch times and intervals, and specifically a re-fetch schedule which tries to adapt the period between consecutive fetches to the period of content changes.

Also, these patches implement checking if the content has changed since last fetching; protocol plugins are also changed to make use of this information, so that if content is unmodified it doesn't have to be fetched and processed.