[HADOOP-16228] Throwing OOM exception for ListStatus (v2) when using S3A with non S3-compatible service

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.1.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Ranith Sardar
• Created: 02/Apr/19 05:08
• Updated: 05/Apr/19 11:24
• Resolved: 05/Apr/19 11:24

Description

 @InterfaceStability.Unstable
  public static final String LIST_VERSION = "fs.s3a.list.version";
  @InterfaceStability.Unstable
  public static final int DEFAULT_LIST_VERSION = 2;

If the files in the bucket are more than 1k, it will throw OOM error for V2 version.

Issue Links

• is broken by: HADOOP-13421 Switch to v2 of the S3 List Objects API in S3A