[HADOOP-3733] "s3:" URLs break when Secret Key contains a slash, even if encoded

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.17.1, 2.0.2-alpha
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fs/s3
• Hadoop Flags: Reviewed
• Release Note: Allows userinfo component of URI authority to contain a slash (escaped as %2F). Especially useful for accessing AWS S3 with distcp or hadoop fs.
• Assignee: Steve Loughran
• Reporter: Stuart Sierra
• Created: 09/Jul/08 20:03
• Updated: 28/Aug/18 21:34
• Resolved: 16/Jun/16 18:16

Description

When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F.

Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
And your bucket is called "mybucket"

You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv

But this doesn't work:

$ bin/hadoop distcp file:///source  s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket
org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden
        at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
...
With failures, global counters are inaccurate; consider running with -i
Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
        at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
...

Issue Links

• breaks: HADOOP-14439 regression: secret stripping from S3x URIs breaks some downstream code
• breaks: HADOOP-13287 TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.
• breaks: HADOOP-14114 S3A can no longer handle unencoded + in URIs
• breaks: SPARK-20799 Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL
• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• is duplicated by: HADOOP-10511 s3n:// incorrectly handles URLs with secret keys that contain a slash
• is duplicated by: HADOOP-14022 S3 connections fails if in-URI AWS secret key contains '+' and '/' both
• is related to: HADOOP-14833 Remove s3a user:secret authentication
• is related to: HDFS-13 filenames with ':' colon throws java.lang.IllegalArgumentException