Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3174

HdfsSink AWS S3A authentication does not work on JDK 8

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Flume writing to S3A with the following Hdfs Sink configuration fails with AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden...

      a1.sinks = k1
      a1.sinks.k1.channel = c1
      a1.sinks.k1.type = hdfs
      a1.sinks.k1.hdfs.path = s3a://testflume/logs
      

      AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are provided either in flume-env or in core-site.xml and running "hdfs dfs -ls s3a://testflume/logs" works properly.

      The cause and the fix is documented in hadoop-aws/index.md

      A change in the Java 8 JVM broke some of the toString() string generation of Joda Time 2.8.0, which stopped the Amazon S3 client from being able to generate authentication headers suitable for validation by S3.

      Fix: Make sure that the version of Joda Time is 2.8.1 or later, or use a new version of Java 8.

      Tested that authentication is successful with

      • JDK 7
      • JDK 8 + joda-time updated to v2.9.6.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                marcellhegedus Marcell Hegedus
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: