[NUTCH-3044] Generator: NPE when extracting the host part of a URL fails

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.20
• Fix Version/s: 1.21
• Component/s: generator
• Assignee: Unassigned
• Reporter: Sebastian Nagel
• Created: 25/Apr/24 15:28
• Updated: 28/Apr/24 17:29

Description

When extracting the host part of a URL fails, the Generator job fails because of a NPE in the SelectorReducer. This issue is reproducible if the CrawlDb contains an malformed URL, for example, a URL with an unsupported scheme (smb://).

Caused by: java.lang.NullPointerException
  at org.apache.nutch.crawl.Generator$SelectorReducer.reduce(Generator.java:439)
  at org.apache.nutch.crawl.Generator$SelectorReducer.reduce(Generator.java:300)

Issue Links

• links to: GitHub Pull Request #815