[CONNECTORS-214] Add post-extraction inclusions and exclusions into the web connector

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: ManifoldCF 0.1, ManifoldCF 0.2
• Fix Version/s: ManifoldCF 0.3
• Component/s: Web connector
• Assignee: Karl Wright
• Reporter: Erlend Garåsen
• Created: 23/Jun/11 12:09
• Updated: 24/Jul/11 22:24
• Resolved: 24/Jul/11 22:24

Description

If html files are excluded for a job, links in these files will not be followed. If we add inclusion and exclusion filters based on post-extraction, it will be possible to fetch only certain types of documents, such as PDFs.