[NUTCH-2935] DeduplicationJob: failure on URLs with invalid percent encoding

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.18
• Fix Version/s: 1.19
• Component/s: crawldb
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 14/Jan/22 09:11
• Updated: 13/Mar/24 14:51
• Resolved: 17/Jan/22 18:59

Description

The DeduplicationJob may fail with an IllegalArgumentException on invalid percent encodings in URLs:

2021-11-25 04:36:41,747 INFO mapreduce.Job: Task Id : attempt_1637669672674_0018_r_000193_0, Status : FAILED
Error: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 0 in: "YR"
        at java.base/java.net.URLDecoder.decode(URLDecoder.java:232)
        at java.base/java.net.URLDecoder.decode(URLDecoder.java:142)
        at org.apache.nutch.crawl.DeduplicationJob$DedupReducer.getDuplicate(DeduplicationJob.java:211)
...
Exception in thread "main" java.lang.RuntimeException: Crawl job did not succeed, job status:FAILED, reason: Task failed task_1637669672674_0018_r_000193
Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0 killedReduces: 0

The IllegalArgumentException should be caught, logged and if only one of the two URLs with duplicated content is invalid, it should be flagged as duplicate while the valid URL "survives".

Issue Links

• links to: GitHub Pull Request #723