[HIVE-23158] Optimize S3A recordReader policy for Random IO formats

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Flags: Patch
• Assignee: Panagiotis Garefalakis
• Reporter: Panagiotis Garefalakis
• Created: 08/Apr/20 18:39
• Updated: 17/Nov/22 08:50
• Resolved: 12/Apr/20 20:32

Description

S3A filesystem client (inherited by Hadoop) supports the notion of input policies.
These policies tune the behaviour of HTTP requests that are used for reading different filetypes such as TEXT or ORC.

For formats such as ORC and Parquet that do a lot of seek operations, there is an optimized RANDOM mode that reads files only partially instead of fully (default).

I am suggesting to add some extra logic as part of HiveInputFormat to make sure we optimize RecordReader requests for random IO when data is stored on S3A using formats such as ORC or Parquet.

Issue Links

• is fixed by: HIVE-24225 FIX S3A recordReader policy selection
• relates to: HIVE-23393 LLapInputFormat reader policy for Random IO formats
• links to: GitHub Pull Request #972