[HADOOP-11687] Ignore x-* and response headers when copying an Amazon S3 object

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fs/s3
• Assignee: Harsh J
• Reporter: Denis Jannot
• Created: 08/Mar/15 10:17
• Updated: 10/Sep/18 12:29
• Resolved: 01/Apr/16 09:07

Description

The EMC ViPR/ECS object storage platform uses proprietary headers starting by x-emc-* (like Amazon does with x-amz-*).
Headers starting by x-emc-* should be included in the signature computation, but it's not done by the Amazon S3 Java SDK (it's done by the EMC S3 SDK).

When s3a copy an object it copies all the headers, but when the object includes x-emc-* headers, it generates a signature mismatch.

Removing the x-emc-* headers from the copy would allow s3a to be compatible with the EMC ViPR/ECS object storage platform.

Removing the x-* which aren't x-amz-* headers from the copy would allow s3a to be compatible with any object storage platform which is using proprietary headers

Issue Links

• contains: HADOOP-12970 Intermittent signature match failures in S3AFileSystem due connection closure
• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• is duplicated by: HADOOP-12970 Intermittent signature match failures in S3AFileSystem due connection closure
• is related to: HADOOP-15735 backport HADOOP-11687 for hadoop-aws 2.7.x