[NUTCH-2555] URL normalization problem: path not starting with a '/'

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14
• Fix Version/s: 1.15
• Assignee: Unassigned
• Reporter: Gerard Bouchar
• Created: 09/Apr/18 14:32
• Updated: 01/Oct/19 14:30
• Resolved: 12/Jun/18 19:05

Description

When an URL does not have a path but has GET parameters (for instance 'http://example.com?a=1') it should be normalized to add a '/' at the beginning of the path (giving http://example.com/?a=1). Our logs show that non-normalized URLs reach protocol-http, which then uses URL::getFile() to get the path, and tries to send an invalid HTTP request:

GET ?a=1 HTTP/1.0

instead of

GET /?a=1 HTTP/1.0

 

Example URL for which this poses a problem: http://news.fx678.com?171