[NUTCH-1242] Allow disabling of URL Filters in ParseSegment

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5
• Patch Info: Patch Available
• Assignee: Markus Jelsma
• Reporter: Edward Drapkin
• Created: 04/Jan/12 22:31
• Updated: 22/May/13 03:53
• Resolved: 31/Jan/12 15:25

Description

Right now, the ParseSegment job does not allow you to disable URL filtration. For reasons that aren't worth explaining, I need to do this, so I enabled this behavior through the use of a boolean configuration value "parse.filter.urls" which defaults to true.

I've attached a simple, preliminary patch that enables this behavior with that configuration option. I'm not sure if it should be made a command line option or not.