[HADOOP-13421] Switch to v2 of the S3 List Objects API in S3A

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 3.0.0-beta1
• Component/s: fs/s3
• Release Note: S3A now defaults to using the "v2" S3 list API, which speeds up large-scale path listings. Non-AWS S3 implementations may not support this API: consult the S3A documentation on how to revert to the v1 API.
• Assignee: Aaron Fabbri
• Reporter: Steven K. Wong
• Created: 26/Jul/16 00:48
• Updated: 26/Mar/20 16:41
• Resolved: 08/Sep/17 14:10

Description

Unlike version 1 of the S3 List Objects API, version 2 by default does not fetch object owner information, which S3A doesn't need anyway. By switching to v2, there will be less data to transfer/process. Also, it should be more robust when listing a versioned bucket with "a large number of delete markers" (according to AWS).

Methods in S3AFileSystem that use this API include:

• getFileStatus(Path)
• innerDelete(Path, boolean)
• innerListStatus(Path)
• innerRename(Path, Path)

Requires AWS SDK 1.10.75 or later.

Issue Links

• breaks: HADOOP-16228 Throwing OOM exception for ListStatus (v2) when using S3A with non S3-compatible service
• depends upon: HADOOP-13050 Upgrade to AWS SDK 1.11.45
• is depended upon by: HADOOP-14710 Uber-JIRA: Support AWS Snowball
• is related to: HADOOP-13811 s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
• is related to: HADOOP-16942 S3A creating folder level delete markers