[HADOOP-15364] Add support for S3 Select to S3A

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.3.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 04/Apr/18 22:19
• Updated: 12/Feb/19 14:04
• Resolved: 12/Feb/19 14:04

Description

Expect a PoC patch for this in a couple of days;

• it'll depend on an SDK update to work, plus a couple of of other minor changes
• Adds command line option too

  hadoop s3guard select -header use -compression gzip -limit 100 s3a://landsat-pds/scene_list.gz" \
  "SELECT s.entityId FROM S3OBJECT s WHERE s.cloudCover = '0.0' "
  

For wider use we'll need to implement the HADOOP-15229 so that callers can pass down the expression along with any other parameters

Issue Links

• depends upon: HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail"
• depends upon: HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.
• depends upon: HADOOP-15606 FSDataInputStream/FSInputStream to implement getFileLength()
• is depended upon by: MAPREDUCE-7182 MapReduce input format/record readers to support S3 select queries
• is part of: HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.