[NUTCH-243] Some meta-refresh urls get ignored due to matching regular expression

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Duplicate
• Affects Version/s: 0.8
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Dennis Kubes
• Created: 05/Apr/06 05:35
• Updated: 17/Mar/08 16:49
• Resolved: 17/Mar/08 16:49

Description

On fetching of pages with meta-refresh tags the url is taken at face value without any filtering. Some urls, such as those used by struts return with a jsessionid or with query strings. Examples are:

http://www.somesite.com;jsessionid=3123123412ADBE3344...
http://www.somesite.com?querystring=value

The RegexURLFilter will match these urls according to the following regex inside of the regex-urlfilter.txt file:

-[?*!@=]

Should these urls be cleaned up to allow processing and not match the previous URL filter or should they be ignored as they currently are?

Issue Links

• is part of: NUTCH-255 Regular Expression for RegexUrlNormalizer to remove jsessionid