[DROIDS-58] Implement a filter mechanism that allow intecepting every stage of a crawling process

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.1.0
• Fix Version/s: 0.3.0
• Assignee: Unassigned
• Reporter: Mingfai Ma
• Created: 17/Jul/09 10:14
• Updated: 15/Jun/12 02:46

Description

refer to this:
http://mail-archives.apache.org/mod_mbox/incubator-droids-dev/200906.mbox/[e-mail]%3E

assume the process is
1. poll a link from queue
2. fetch entity
3. parse entity
4. extract outlinks

we provide a mechanism to intercept the process in every stage. e.g. a LinkFilter has a "public T polled(T link);" interface, any filter may reject or transform a Link polled from the queue. similar logic applies to fetching, parsing, and extracting (outlink)

Issue Links

• relates to: DROIDS-56 Change the TaskQueue interface to java.util.Queue