[NUTCH-706] Url regex normalizer: default pattern for session id removal not to match "newsId"

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.6, 2.2
• Assignee: Unassigned
• Reporter: Meghna Kukreja
• Created: 27/Feb/09 18:47
• Updated: 22/May/13 03:54
• Resolved: 10/Oct/12 21:14

Description

Hey,

I encountered the following problem while trying to crawl a site using
nutch-trunk. In the file regex-normalize.xml, the following regex is
used to remove session ids:

<pattern>([;_]?((?i)l|j|bv_)?((?i)sid|phpsessid|sessionid)=.*?)(?|&|#|$)</pattern>.

This pattern also transforms a url, such as,
"&newsId=2000484784794&newsLang=en" into "&new&newsLang=en" (since it
matches 'sId' in the 'newsId'), which is incorrect and hence does not
get fetched. This expression needs to be changed to prevent this.

Thanks,
Meghna

Issue Links

• is duplicated by: NUTCH-1328 a problem with regex-normalize.xml