wrong regular expression hang reduce process for ever

• Key: NUTCH-233
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Unassigned
• Reporter: Stefan Groschupf
• Affects Version/s: 0.8
• Fix Version/s: 0.9.0

Description

Looks like that the expression ".(/.+?)/.?\1/.*?\1/" in regex-urlfilter.txt wasn't compatible with java.util.regex that is actually used in the regex url filter.
May be it was missed to change it when the regular expression packages was changed.
The problem was that until reducing a fetch map output the reducer hangs forever since the outputformat was applying the urlfilter a url that causes the hang.
060315 230823 task_r_3n4zga at java.lang.Character.codePointAt(Character.java:2335)
060315 230823 task_r_3n4zga at java.util.regex.Pattern$Dot.match(Pattern.java:4092)
060315 230823 task_r_3n4zga at java.util.regex.Pattern$Curly.match1(Pattern.java:

I changed the regular expression to ".*(/[^/]+)/[^/]+\1/[^/]+\1/" and now the fetch job works. (thanks to Grant and Chris B. helping to find the new regex)
However may people can review it and can suggest improvements, since the old regex would match :
"abcd/foo/bar/foo/bar/foo/" and so will the new one match it also. But the old regex would also match :
"abcd/foo/bar/xyz/foo/bar/foo/" which the new regex will not match.

Comments

[Jerome Charron]

Stefan,

I have created a small unit test for urlfilter-regexp and I doesn't notice any incompatibility in java.util.regex with this regexp. Could you please provide the urls that cause problem so that I can add them to me unit tests.
Thanks

Jérôme

[Stefan Groschupf]

Sorry, I haven't such url since it happens until reducing a fetch. Reducing provides no logging and map data will be deleted if the job fails because a timeout.

[Stefan Groschupf]

I think this should be fixed in .8 too, since everybody that does real whole web crawl with over a 100 Mio pages will run into this problem. The problems are for example from spam bot generated urls.

[Otis Gospodnetic]

I haven't noticed this regexp being a problem so far either, but maybe I've just been lucky not to have run into bot-trap site yet. Is this still a problem for you, Stefan?

[Stefan Groschupf]

Hi Otis,
yes for a serious whole web crawl I need to change this reg ex first.
It only hangs with some random urls that for example comes from link farms the crawler runs into.

Could I suggest that this change, from ".(/.?)/.?\1/.?\1/" to ".(/[^/])/[^/]+\1/[^/]+\1/" be committed to at least trunk for the time being.

I recently created a segment with 1M urls exactly, I ran the fetch and it did indeed stall on the reduce part of the operation due to the regex filter. This was verified with a thread dump (kill -3 <pid>) on FreeBSD.

I then made the suggested change in the config file and re-fetched the exact same segment. It completed without issue.

I'm aware we might be losing some filtering functionality with this new expression, but is it not better then knowing there is always the chance your whole-web crawl fetch will fail because of this?

[Dennis Kubes]

The new regex has been added to both the regex-urlfilter.txt and the crawl-urlfilter.txt files.

[Dennis Kubes]

Issue closed