[NUTCH-2950] UpdateHostDb: performance improvements

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.18
• Fix Version/s: 1.19
• Component/s: hostdb
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 19/May/22 14:59
• Updated: 13/Mar/24 14:51
• Resolved: 24/May/22 12:55

Description

This issue addresses a couple of performance improvements when creating the HostDb:

• avoid needless conversions between hostname and URL
• improvements of HostDb serialization (write and read)
• parametrize logging and log less on level INFO
• do not create DNS resolver threads if DNS look-ups are not requested by command-line options

A patch/PR is ready. Depending on the chosen command-line options, a 10-20% speed-up should be visible if DNS look-ups, normalization and filtering are off.

Issue Links

• links to: GitHub Pull Request #731