[NUTCH-249] black- white list url filtering

• Type: Improvement
• Status: Closed
• Priority: Trivial
• Resolution: Won't Fix
• Affects Version/s: 0.8
• Component/s: fetcher
• Assignee: Dennis Kubes
• Reporter: Stefan Groschupf
• Created: 17/Apr/06 20:59
• Updated: 01/Apr/11 15:03
• Resolved: 01/Apr/11 15:03

Description

Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.