[CONNECTORS-104] Make it easier to limit a web crawl to a single site

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: ManifoldCF 0.1
• Component/s: Web connector
• Assignee: Karl Wright
• Reporter: Jack Krupansky
• Created: 07/Sep/10 21:07
• Updated: 23/Aug/18 22:27
• Resolved: 08/Sep/10 13:11

Description

Unless the user explicitly enters an include regex carefully, a web crawl can quickly get out of control and start crawling the entire web when all the user may really want is to crawl just a single web site or portion thereof. So, it would be preferable if either by default or with a simple button the crawl could be limited to the seed web site(s).