[NUTCH-2378] ChildFirst plugin classloader

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1, 1.13
• Fix Version/s: 2.4, 1.14
• Component/s: plugin
• Patch Info: Patch Available
• Flags: Patch
• Assignee: Sebastian Nagel
• Reporter: Jurian Broertjes
• Created: 01/May/17 15:42
• Updated: 13/Mar/24 14:50
• Resolved: 18/Aug/17 14:34

Description

While working on upgrading the indexer-elastic plugin from 2.x to 5.x, I ran into several nasty runtime dependency issues (both local and on Hadoop). After seeking help on the mailing list, I still was unable to resolve these issues and after digging further, decided to try a different plugin classloader strategy.

The normal classloader delegates class loading requests to it's parent classloader. This can cause all sorts of nasty runtime dependency version conflicts (jar hell, version conflicts), since the plugin's own classloader gets queried last. The child-first classloader approach tries to load a class from the plugin's dependencies first and when unavailable, delegates to it's parent classloader. This fixed the issues I had.

The new approach can give runtime LinkageErrors, but these are easily resolvable (see the patch for a few examples)

I've tested the new loader a bit and am curious about others' findings.

Issue Links

• contains: NUTCH-2071 A parser failure on a single document may fail crawling job if parser.timeout=-1
• contains: NUTCH-2316 Library conflict with Parser-Tika Plugin and Lib Folder
• is required by: NUTCH-2380 indexer-elastic version upgrade to 5.3.0