[NUTCH-2611] Add line-breaks when parsing HTML block-level elements

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.14
• Fix Version/s: 1.15
• Component/s: parser
• Assignee: Unassigned
• Reporter: Yossi Tamari
• Created: 25/Jun/18 11:38
• Updated: 01/Oct/19 14:28
• Resolved: 28/Jun/18 09:08

Description

Currently, the HTML and Tika parser only add newlines following text-nodes that contain only whitespaces (e.g </span> <span>), but not based on what the tags are, so for example a </div><div> will not add a new line.

While some applications do not differentiate between a space and a new line, many others see the semantic difference (two following words in the same sentence are "near", but in separate sentences they are not).

I believe adding newlines after block-level HTML elements, while not a panacea, will be an improvement on the current behavior.

NUTCH-2318 is related to this.

Issue Links

• links to: GitHub Pull Request #354