[FLUME-3174] HdfsSink AWS S3A authentication does not work on JDK 8

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.8.0
• Assignee: Unassigned
• Reporter: Marcell Hegedus
• Created: 11/Sep/17 14:32
• Updated: 12/Sep/17 14:30
• Resolved: 12/Sep/17 14:22

Description

Flume writing to S3A with the following Hdfs Sink configuration fails with AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden...

a1.sinks = k1
a1.sinks.k1.channel = c1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = s3a://testflume/logs

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are provided either in flume-env or in core-site.xml and running "hdfs dfs -ls s3a://testflume/logs" works properly.

The cause and the fix is documented in hadoop-aws/index.md

A change in the Java 8 JVM broke some of the toString() string generation of Joda Time 2.8.0, which stopped the Amazon S3 client from being able to generate authentication headers suitable for validation by S3.

Fix: Make sure that the version of Joda Time is 2.8.1 or later, or use a new version of Java 8.

Tested that authentication is successful with

• JDK 7
• JDK 8 + joda-time updated to v2.9.6.

Issue Links

• requires: FLUME-3173 Upgrade joda-time