[DROIDS-77] Be able to modify URL rules while crawler is running

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Component/s: core
• Assignee: Unassigned
• Reporter: Richard Frovarp
• Created: 14/Feb/10 23:19
• Updated: 14/Dec/10 22:18
• Resolved: 14/Dec/10 22:16

Description

It would be nice to be able to modify the URL rules while a crawler is running. This would allow me to dynamically exclude areas from being crawled based on results being returned. Basically I want to look for certain markers inside a page, then not crawl those pages without having update a robots file. Different paths of our site is going to enter into the index from a different method than the main crawl, so I can skip them once I find them.

Having a modifiable filter would allow people to load their rules from places other than a file without having to write their own implementation or extension. I'll try to work up a patch sometime this week.

Issue Links

• duplicates: DROIDS-111 Open the API of RegexURLFilter to other types of input