[NUTCH-1483] Can't crawl filesystem with protocol-file plugin

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.6, 2.1
• Fix Version/s: 2.3, 1.10
• Component/s: protocol

Environment

OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4

• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Rogério Pereira Araújo
• Created: 31/Oct/12 16:14
• Updated: 13/Mar/24 14:51
• Resolved: 04/Nov/14 21:12

Description

I tried to follow the same steps described in this wiki page:

http://wiki.apache.org/nutch/IntranetDocumentSearch

I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:

file:///home/rogerio/Documents/

The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:

org.apache.nutch.protocol.file.FileError: File Error: 404
at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404

Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???

Note: The regex-urlfilter entry only works as expected if I add the entry
+^{file://home/rogerio/Documents/ instead of +}file:///home/rogerio/Documents/ as wiki says.

Issue Links

• duplicates: NUTCH-1076 Solrindex has no documents following bin/nutch solrindex when using protocol-file
• is duplicated by: NUTCH-1076 Solrindex has no documents following bin/nutch solrindex when using protocol-file

Sub-Tasks

• urlnormalizer-regex to keep third slash in file:///path/index.html: Closed
• Regex URL normalizer should remove multiple slashes after file: protocol: Closed
• URLUtil should not add additional slashes for file URLs: Closed
• Protocol-file should treat symbolic links as redirects: Closed