[#NUTCH-279] Additions for regex-normalize

Nutch

Additions for regex-normalize

Created: 22/May/06 08:09 PM Updated: 10/Apr/09 12:29 PM

Return to search

Component/s:

None

Affects Version/s:

0.8

Fix Version/s:

1.0.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
regex-normalize.patch	2006-05-22 08:12 PM	Stefan Neufeind	4 kB
regex-normalize2.patch	2006-07-09 10:32 PM	Stefan Neufeind	4 kB

Issue Links:

Incorporates

This issue incorporates:

~~NUTCH-255~~ Regular Expression for RegexUrlNormalizer to remove jsessionid

Resolution Date:

03/Feb/09 03:16 PM

Description

« Hide

Imho needed:
1) Extend normalize-rules to commonly used session-id's etc.
2) Ship a checker to check rules easily by hand

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Stefan Neufeind added a comment - 22/May/06 08:12 PM

1) Incorporates jsessionid-normalization from ~~NUTCH-255~~
2) Adds further normalizations
3) Adds a commandline-checker. Start with:
bin/nutch org.apache.nutch.net.RegexUrlNormalizerChecker

Stefan Neufeind made changes - 22/May/06 08:12 PM

Field	Original Value	New Value
Attachment		regex-normalize.patch [ 12334408 ]

Stefan Neufeind made changes - 22/May/06 08:13 PM

Link

This issue incorporates ~~NUTCH-255~~ [ ~~NUTCH-255~~ ]

[ Permlink | « Hide ]

Stefan Neufeind added a comment - 09/Jul/06 10:32 PM

New patch with just one session-ID-regex extended (also including . - , now), since I came across those extra chars while used on a common German website (www.bahn.de).

Stefan Neufeind made changes - 09/Jul/06 10:32 PM

Attachment

regex-normalize2.patch [ 12336542 ]

Repository	Revision	Date	User	Message
ASF	#740318	Tue Feb 03 15:12:48 UTC 2009	ab	~~NUTCH-279~~ Additions to urlnormalizer-regex (modified).
				Files Changed
				ADD /lucene/nutch/trunk/src/java/org/apache/nutch/net/URLNormalizerChecker.java MODIFY /lucene/nutch/trunk/CHANGES.txt

Andrzej Bialecki made changes - 03/Feb/09 03:16 PM

Resolution		Fixed [ 1 ]
Fix Version/s		1.0.0 [ 12312443 ]
Assignee		Andrzej Bialecki [ ab ]
Status	Open [ 1 ]	Closed [ 6 ]

[ Permlink | « Hide ]

Andrzej Bialecki added a comment - 03/Feb/09 03:16 PM

Commited with some modifications. All patterns in this patch except one have been added in another commit, the remaining one (-S: ...) IMHO occurs too rarely and the pattern would be too incusive. The checking utility has been rewritten to follow a similar model like URLFilterChecker.

[ Permlink | « Hide ]

Hudson added a comment - 04/Feb/09 04:10 AM

Integrated in Nutch-trunk #714 (See http://hudson.zones.apache.org/hudson/job/Nutch-trunk/714/)
Additions to urlnormalizer-regex (modified).