[NUTCH-1885] Protocol-file should treat symbolic links as redirects

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.9, 2.2.1
• Fix Version/s: 2.3, 1.10
• Component/s: protocol
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Sebastian Nagel
• Created: 31/Oct/14 19:18
• Updated: 13/Mar/24 14:50
• Resolved: 04/Nov/14 21:16

Description

(reported by angela_wang, see NUTCH-1884, [1 and [2)

If a file is a symbolic link or contains a link on it's path:, protocol-file follows the link immediately and returns a Content object with the canonical path (all symbolic links resolved) in field "Location". This may cause

• the Parse object not available under its expected URL (see NUTCH-1884)
• dubious CrawlDatums (status fetched!) in CrawlDb (first URL is a symbolic link to second item):

  file:/var/www/redir_test.html   Version: 7
  Status: 2 (db_fetched)
  ...
  Signature: null
  Metadata: 
          Content-Type=text/html
          _pst_=success(1), lastModified=0
  
  file:/var/www/test.html Version: 7
  Status: 2 (db_fetched)
  ...
  Signature: 50fa8436398f0ecb6b15eaba0574ef23
  Metadata: 
          Content-Type=text/html
          _pst_=success(1), lastModified=0
  

  Because signature is null these will never result in duplicates in index.

Protocol-file should instead explicitly redirect to the link target. This should be the default, optionally we could add a property to restore the old behavior.

Should not be difficult to resolve: FileResponse already has status "redirect" for symlinks, but File.getProtocolOutput() then resolves the links internally. So we just need to return a redirect response before links are resolved/followed.