[NUTCH-2689] Speed up urlfilter-regex and urlfilter-automaton

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Affects Version/s: 1.15
• Fix Version/s: 1.16
• Component/s: plugin
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 22/Jan/19 13:45
• Updated: 28/Jan/21 13:55
• Resolved: 29/Jan/19 10:32

Description

The unit tests of urlfilter-regex and urlfilter-automaton include a benchmark. After playing and benchmarking modifications the following changes seem to significantly improve the performance:

• do not extract host and domain name from the URL if not needed (no host/domain-specific rules used, cf. NUTCH-1838)
• use non-capturing groups if possible
• use (?i) to make the patterns case insensitive and remove uppercase variants

Issue Links

• links to: GitHub Pull Request #432