Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15220 Über-jira: S3a phase V: Hadoop 3.2 features
  3. HADOOP-14439

regression: secret stripping from S3x URIs breaks some downstream code

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 2.8.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      Spark 2.1

    • Target Version/s:
    • Release Note:
      Hide
      We have restored the retention of a username:secret in an s3a: URL if you authenticate with S3 using embedded URI secrets. This is because some applications using strings to marshall the URLs were breaking. However, using secrets in this way is dangerous as it will end up in logs. It will be unsupported in Hadoop 3. To use different credentials in different buckets, move to per-bucket configuration
      Show
      We have restored the retention of a username:secret in an s3a: URL if you authenticate with S3 using embedded URI secrets. This is because some applications using strings to marshall the URLs were breaking. However, using secrets in this way is dangerous as it will end up in logs. It will be unsupported in Hadoop 3. To use different credentials in different buckets, move to per-bucket configuration

      Description

      Surfaced in SPARK-20799

      Spark is listing the contents of a path with getFileStatus(path), then looking up the path value doing a lookup of the contents.

      Apparently the lookup is failing to find files if you have a secret in the key, s3a://key:secret@bucket/path.

      Presumably this is because the stripped values aren't matching.

        Attachments

        1. HADOOP-14439-02.patch
          4 kB
          Vinayakumar B
        2. HADOOP-14439-01.patch
          4 kB
          Vinayakumar B

          Issue Links

            Activity

              People

              • Assignee:
                vinayakumarb Vinayakumar B
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: