Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3733

"s3:" URLs break when Secret Key contains a slash, even if encoded



    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.17.1, 2.0.2-alpha
    • 2.8.0, 3.0.0-alpha1
    • fs/s3
    • None
    • Reviewed
    • Allows userinfo component of URI authority to contain a slash (escaped as %2F). Especially useful for accessing AWS S3 with distcp or hadoop fs.


      When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F.

      Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
      And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
      And your bucket is called "mybucket"

      You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv

      But this doesn't work:

      $ bin/hadoop distcp file:///source  s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
      08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
      08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
      08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket
      org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden
              at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
      With failures, global counters are inaccurate; consider running with -i
      Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
              at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)


        1. HADOOP-3733-branch-2-007.patch
          44 kB
          Steve Loughran
        2. HADOOP-3733-branch-2-006.patch
          43 kB
          Steve Loughran
        3. HADOOP-3733-branch-2-005.patch
          43 kB
          Steve Loughran
        4. HADOOP-3733-branch-2-004.patch
          40 kB
          Steve Loughran
        5. HADOOP-3733-branch-2-003.patch
          40 kB
          Steve Loughran
        6. HADOOP-3733-branch-2-002.patch
          32 kB
          Steve Loughran
        7. HADOOP-3733-branch-2-001.patch
          31 kB
          Steve Loughran
        8. HADOOP-3733-20130223T011025Z.patch
          7 kB
          David Chaiken
        9. HADOOP-3733.patch
          7 kB
          David Chaiken
        10. hadoop-3733.patch
          2 kB
          Paul Butler

        Issue Links



              stevel@apache.org Steve Loughran
              stuartsierra Stuart Sierra
              9 Vote for this issue
              38 Start watching this issue