[NUTCH-2182] Make reverseUrlDirs file dumper option hash the URL for consistency

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11
• Fix Version/s: 1.12
• Component/s: tool
• Assignee: Michael Joyce
• Reporter: Michael Joyce
• Created: 08/Dec/15 23:01
• Updated: 13/Mar/24 14:50
• Resolved: 16/Dec/15 22:10

Description

At the moment the "reverseUrlDirs" option for FileDumper is terribly brittle and fails on a fair number of edge cases. A more robust way to handle the reverse URL approach to dumping a file is to reverse the server part and hash the URL to use as the file name. This gives us a nice split of files while avoiding a number of likely classes that causes dumps to fail.

Issue Links

• contains: NUTCH-2187 Change FileDumper SHAs to all uppercase