[NUTCH-2429] Fix Plugin System to allow protocol plugins to bundle their URLStreamHandlers

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14
• Fix Version/s: 1.19
• Component/s: commoncrawl

Environment

Tested on both Nutch 1.13 and 1.14 in Ubuntu Linux with OpenJDK 1.8.

• Assignee: Lewis John McGibbney
• Reporter: Hiran Chaudhuri
• Created: 22/Sep/17 10:22
• Updated: 13/Mar/24 14:51
• Resolved: 08/Jan/22 04:08

Description

While trying to use the protocol-smb plugin (which is not part of the Nutch distribution) I realized there are four steps to successfully make use of a protocol plugin:

1 - put the artifact into the plugins directory
2 - modify Nutch configuration files to allow smb:// urls plus include the plugin to the loaded list
3 - extract jcifs.jar and place it on the system classpath
4 - run nutch with the correct system property

While steps 1 and 2 seem obvious, 3 and 4 require knowledge of plugin internals which does not feel right for nutch and plugin users. Even more, the jcifs.jar would exist twice on the classpath and could even cause further problems during runtime.

Issue Links

• causes: NUTCH-2936 Early registration of URL stream handlers provided by plugins may fail Hadoop jobs running in distributed mode if protocol-okhttp is used
• causes: NUTCH-2949 Tasks of a multi-threaded map runner may fail because of slow creation of URL stream handlers
• is related to: NUTCH-714 Need a SFTP and SCP Protocol Handler
• is related to: NUTCH-427 protocol-smb: plugin protocol implementing the CIFS/SMB protocol. This protocol allows Nutch to crawl Microsoft Windows Shares remotely using the CIFS/SMB protocol implmentation.
• links to: GitHub Pull Request #222
• links to: GitHub Pull Request #720
• links to: related discussion on user@nutch