[HADOOP-15735] backport HADOOP-11687 for hadoop-aws 2.7.x

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.7.7
• Fix Version/s: 2.7.8
• Component/s: fs/s3
• Assignee: MunShik JOUNG
• Reporter: MunShik JOUNG
• Created: 10/Sep/18 01:39
• Updated: 21/Sep/18 09:41
• Resolved: 21/Sep/18 09:41

Description

HADOOP-12970 Intermitent SignatureDoesNotMatch Exception is fixed by HADOOP-11687.

And from hadoop-aws >= 2.8, HADOOP-11687 is applied. but Spark 2.3.1 is still using hadoop-aws 2.7.7.

I was stumpled upon Intermitent SignatureDoesNotMatch Exception with s3 with spark 2.3.1. So I guess backporting might benefit some spark users. In my situation, SignatureDoesNotMatch was reliably reproducible (4 out of 8 attemp). And after using patched version symptom was not seen (0 out of 10)

Issue Links

• relates to: HADOOP-11687 Ignore x-* and response headers when copying an Amazon S3 object