[YARN-10444] Node Manager to use openFile() with whole-file read policy for localizing files.

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.5
• Component/s: nodemanager
• Target Version/s: 3.3.9
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 18/Sep/20 18:34
• Updated: 02/Mar/23 13:21
• Resolved: 02/Mar/23 13:21

Description

HADOOP-16202 adds standard options for declaring the read/seek
Policy when reading a file. These should be set to sequential IO
When localising resources, so that if the default/cluster settings
For a file system are optimized for random IO, artifact downloads
are still read at the maximum speed possible (one big GET to the EOF).

Most of this happens in hadoop-common, but some tuning of FSDownload
can assist

• tar/jar download must also be sequential
• if the FileStatus is passed around, that can be used
  in the open request to skip checks when loading the file.

Together this can save 3 HEAD requests per resource, with the sequential
IO avoiding any splitting of the big read into separate block GETs

Issue Links

• depends upon: HADOOP-16202 Enhance openFile() for better read performance against object stores
• links to: https://github.com/apache/hadoop/pull/2168