Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: fs/s3
    • Labels:
    • Target Version/s:

      Description

      Use a config setting to allow customizing the used AWS region.

      It also enables using a custom url pointing to an S3-compatible object store.

      1. JIRA-11261.patch
        6 kB
        Thomas Demoor
      2. HADOOP-11261-2.patch
        6 kB
        Thomas Demoor
      3. HADOOP-11261-3.patch
        6 kB
        Thomas Demoor

        Issue Links

          Activity

          Hide
          thodemoor Thomas Demoor added a comment -

          Change proposed by Takenori Sato in the comments of HADOOP-10400.

          Additions:

          • possible URI malformed exception caught and logged as error
          • added a documented test that ran succesfully against buckets in Ireland and Oregon regions

          In order to design a meaningful test I had to expose a public getter for the AmazonS3Client in S3AFileSystem To indicate to other developers that manipulating this object can cause errors a "Here be dragons" warning was added as a comment Probably there is a better and standardized way (annotation?) to indicate this in Hadoop... Anyone?

          Note: related to HADOOP-11171 as the test class is shared. The last one to get accepted will probably need to be diffed vs trunk again.

          Show
          thodemoor Thomas Demoor added a comment - Change proposed by Takenori Sato in the comments of HADOOP-10400 . Additions: possible URI malformed exception caught and logged as error added a documented test that ran succesfully against buckets in Ireland and Oregon regions In order to design a meaningful test I had to expose a public getter for the AmazonS3Client in S3AFileSystem To indicate to other developers that manipulating this object can cause errors a "Here be dragons" warning was added as a comment Probably there is a better and standardized way (annotation?) to indicate this in Hadoop... Anyone? Note: related to HADOOP-11171 as the test class is shared. The last one to get accepted will probably need to be diffed vs trunk again.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Guava's @VisibleForTesting is the secret annotation. Making it package scoped and putting tests in the same package keeps things more locked down

          Show
          stevel@apache.org Steve Loughran added a comment - Guava's @VisibleForTesting is the secret annotation. Making it package scoped and putting tests in the same package keeps things more locked down
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Quick glance over

          1. invalid arguments must be propagated. When wouldn't you want an erroneous configuration URL to be ignored?
          2. WIP styleguide that covers some of the test code : https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md
          3. tests should refer to the new s3a config options as constants, rather than cut-and-paste strings.
          Show
          stevel@apache.org Steve Loughran added a comment - Quick glance over invalid arguments must be propagated. When wouldn't you want an erroneous configuration URL to be ignored? WIP styleguide that covers some of the test code : https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md tests should refer to the new s3a config options as constants, rather than cut-and-paste strings.
          Hide
          thodemoor Thomas Demoor added a comment -

          Thanks, Steve. Patch 2 reflects your fixes.

          Invalid arguments must be propagated. When wouldn't you want an erroneous configuration URL to be ignored?

          Wow, this will haunt me in my dreams for quite some time

          WIP styleguide that covers some of the test code : https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md

          This was exactly what I was looking for. Adheres to much higher standards than the (now obviously legacy) s3n test code I was trying to mimic.

          Show
          thodemoor Thomas Demoor added a comment - Thanks, Steve. Patch 2 reflects your fixes. Invalid arguments must be propagated. When wouldn't you want an erroneous configuration URL to be ignored? Wow, this will haunt me in my dreams for quite some time WIP styleguide that covers some of the test code : https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md This was exactly what I was looking for. Adheres to much higher standards than the (now obviously legacy) s3n test code I was trying to mimic.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12679240/HADOOP-11261-2.patch
          against trunk revision 80d7d18.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5042//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5042//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679240/HADOOP-11261-2.patch against trunk revision 80d7d18. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5042//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5042//console This message is automatically generated.
          Hide
          thodemoor Thomas Demoor added a comment -

          Refactored tests to use the (recently introduced) S3ATestUtils.

          Show
          thodemoor Thomas Demoor added a comment - Refactored tests to use the (recently introduced) S3ATestUtils.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12685614/HADOOP-11261-3.patch
          against trunk revision 1b3bb9e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 7 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5188//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5188//artifact/patchprocess/newPatchFindbugsWarningshadoop-aws.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5188//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685614/HADOOP-11261-3.patch against trunk revision 1b3bb9e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 7 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5188//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5188//artifact/patchprocess/newPatchFindbugsWarningshadoop-aws.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5188//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          All tests pass, reviewed code looks good

          +1, committing

          Thomas: you are going to need to write a site documentation patch which covers this and the other configuration parameters you've got in pending changes

          Show
          stevel@apache.org Steve Loughran added a comment - All tests pass, reviewed code looks good +1, committing Thomas: you are going to need to write a site documentation patch which covers this and the other configuration parameters you've got in pending changes
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6876 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6876/)
          HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6876 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6876/ ) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/)
          HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/ ) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/809/)
          HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/809/ ) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/)
          HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/ ) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/)
          HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/ ) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/)
          HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/ ) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/)
          HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/ ) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

            People

            • Assignee:
              thodemoor Thomas Demoor
              Reporter:
              thodemoor Thomas Demoor
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development