[NUTCH-603] Add more default url normalizations

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0

Environment

All

• Patch Info: Patch Available
• Assignee: Dennis Kubes
• Reporter: Dennis Kubes
• Due: 08/Feb/08
• Created: 05/Feb/08 16:58
• Updated: 10/Apr/09 12:29
• Resolved: 14/Feb/08 22:17

Description

By default the regex-urlnormalizers only remove PHPSESSID strings. I propose adding in more default url normalizers including expressions for removing different types of session ids, removing default pages, remvoing interpage links, and cleaning up url strings. The point of these expressions is to decrease the number of duplicate urls that are being stored and scored in the crawl database and being fetched.