Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12963

Allow using path style addressing for accessing the s3 endpoint

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
    • Target Version/s:
    • Release Note:
      Add new flag to allow supporting path style addressing for s3a

      Description

      There is no ability to specify using path style access for the s3 endpoint. There are numerous non-amazon implementations of storage that support the amazon API's but only support path style access such as Cleversafe and Ceph. Additionally in many environments it is difficult to configure DNS correctly to get virtual host style addressing to work

      1. HADOOP-12963-001.patch
        8 kB
        Stephen Montgomery
      2. HADOOP-12963-002.patch
        8 kB
        Stephen Montgomery
      3. HADOOP-12963-1.patch
        8 kB
        Stephen Montgomery
      4. hdfs-8728.patch.2
        3 kB
        Andrew Baptist

        Issue Links

          Activity

          Hide
          abaptist Andrew Baptist added a comment -

          add new parameter fs.s3a.path.style.access

          Show
          abaptist Andrew Baptist added a comment - add new parameter fs.s3a.path.style.access
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 0s Site still builds.
          -1 checkstyle 0m 20s The applied patch generated 3 new checkstyle issues (total was 62, now 64).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 0m 45s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 tools/hadoop tests 0m 11s Tests passed in hadoop-aws.
              43m 12s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744112/hdfs-8728.patch
          Optional Tests javadoc javac unit findbugs checkstyle site
          git revision trunk / c9dd2ca
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11614/artifact/patchprocess/diffcheckstylehadoop-aws.txt
          hadoop-aws test log https://builds.apache.org/job/PreCommit-HDFS-Build/11614/artifact/patchprocess/testrun_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11614/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11614/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 3m 0s Site still builds. -1 checkstyle 0m 20s The applied patch generated 3 new checkstyle issues (total was 62, now 64). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 0m 45s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 tools/hadoop tests 0m 11s Tests passed in hadoop-aws.     43m 12s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744112/hdfs-8728.patch Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / c9dd2ca checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11614/artifact/patchprocess/diffcheckstylehadoop-aws.txt hadoop-aws test log https://builds.apache.org/job/PreCommit-HDFS-Build/11614/artifact/patchprocess/testrun_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11614/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11614/console This message was automatically generated.
          Hide
          abaptist Andrew Baptist added a comment -

          No easy way to add unit tests for this - this requires integration with an external system to test.

          Show
          abaptist Andrew Baptist added a comment - No easy way to add unit tests for this - this requires integration with an external system to test.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 patch 0m 1s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
          0 pre-patch 19m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 1s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 9m 54s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 2s Site still builds.
          -1 checkstyle 0m 19s The applied patch generated 1 new checkstyle issues (total was 62, now 62).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 0m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 tools/hadoop tests 0m 13s Tests passed in hadoop-aws.
              44m 16s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744236/hdfs-8728.patch.2
          Optional Tests javadoc javac unit findbugs checkstyle site
          git revision trunk / 98e5926
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11625/artifact/patchprocess/diffcheckstylehadoop-aws.txt
          hadoop-aws test log https://builds.apache.org/job/PreCommit-HDFS-Build/11625/artifact/patchprocess/testrun_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11625/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11625/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 patch 0m 1s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. 0 pre-patch 19m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 1s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 9m 54s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 site 3m 2s Site still builds. -1 checkstyle 0m 19s The applied patch generated 1 new checkstyle issues (total was 62, now 62). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 tools/hadoop tests 0m 13s Tests passed in hadoop-aws.     44m 16s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744236/hdfs-8728.patch.2 Optional Tests javadoc javac unit findbugs checkstyle site git revision trunk / 98e5926 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11625/artifact/patchprocess/diffcheckstylehadoop-aws.txt hadoop-aws test log https://builds.apache.org/job/PreCommit-HDFS-Build/11625/artifact/patchprocess/testrun_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11625/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11625/console This message was automatically generated.
          Hide
          abaptist Andrew Baptist added a comment -

          After further testing, it appears that the Amazon S3 library is automatically setting this property when the "custom endpoint" is set, so this request is unnecessary and can be closed

          Show
          abaptist Andrew Baptist added a comment - After further testing, it appears that the Amazon S3 library is automatically setting this property when the "custom endpoint" is set, so this request is unnecessary and can be closed
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Since no patch was committed, we should not resolve this as fixed.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Since no patch was committed, we should not resolve this as fixed.
          Hide
          smontgomery Stephen Montgomery added a comment -

          Hi,
          I'd like to re-open this ticket please. I've done some further digging into this and believe that Andrew's original patch is still needed ie using a Hadoop S3A config property flag to "switch on" path style access in the underlying Amazon S3 client. Overriding the custom S3A endpoint has no effect (unless you specifically use an IPv4 address which is more a workaround/hack).

          To force/trick the Amazon S3 client to use old path style access (instead of virtual hosting) you can use dodgy bucket names (eg '..', '.-' in the name, caps etc) and IPv4 addresses for the endpoint - see com.amazonaws.services.s3.AmazonS3Client. configRequest() method - pretty much making sure that the DNS lookups will fail for syntactic reasons.

          I'm happy to update Andrew's original patch and supply a test case, if needed. Like Andrew mentioned, the test case will be of no real benefit as it will just exercising Amazon client functionality. It's also hard to do as the AWS client is pretty inaccessible around confirming the flag has been set.

          Whats the process of re-opening this ticket? What Hadoop branch will this be targeted for ie it looks that 2.8 one has all of the S3A fixes...?

          Thanks,
          Stephen

          Show
          smontgomery Stephen Montgomery added a comment - Hi, I'd like to re-open this ticket please. I've done some further digging into this and believe that Andrew's original patch is still needed ie using a Hadoop S3A config property flag to "switch on" path style access in the underlying Amazon S3 client. Overriding the custom S3A endpoint has no effect (unless you specifically use an IPv4 address which is more a workaround/hack). To force/trick the Amazon S3 client to use old path style access (instead of virtual hosting) you can use dodgy bucket names (eg '..', '.-' in the name, caps etc) and IPv4 addresses for the endpoint - see com.amazonaws.services.s3.AmazonS3Client. configRequest() method - pretty much making sure that the DNS lookups will fail for syntactic reasons. I'm happy to update Andrew's original patch and supply a test case, if needed. Like Andrew mentioned, the test case will be of no real benefit as it will just exercising Amazon client functionality. It's also hard to do as the AWS client is pretty inaccessible around confirming the flag has been set. Whats the process of re-opening this ticket? What Hadoop branch will this be targeted for ie it looks that 2.8 one has all of the S3A fixes...? Thanks, Stephen
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. yes, re-open and supply a new patch. Target branch-2 and trunk; things will get into 2.9 and maybe 2.8
          2. moving to hadoop project,, component fs/s3a and linking to the s3a phase II jIRA
          3. see https://wiki.apache.org/hadoop/HowToContribute for special rules/process for submitting patches to object stores. No details: no review. Sorry.
          4. Like you say: hard to test. you'll have to add some details in the s3a docs to describe this feature
          Show
          stevel@apache.org Steve Loughran added a comment - yes, re-open and supply a new patch. Target branch-2 and trunk; things will get into 2.9 and maybe 2.8 moving to hadoop project,, component fs/s3a and linking to the s3a phase II jIRA see https://wiki.apache.org/hadoop/HowToContribute for special rules/process for submitting patches to object stores. No details: no review. Sorry. Like you say: hard to test. you'll have to add some details in the s3a docs to describe this feature
          Hide
          smontgomery Stephen Montgomery added a comment -

          Uploaded patch - tested against trunk

          Show
          smontgomery Stephen Montgomery added a comment - Uploaded patch - tested against trunk
          Hide
          smontgomery Stephen Montgomery added a comment -

          Submitted patch - see HADOOP-12963-1.patch

          Show
          smontgomery Stephen Montgomery added a comment - Submitted patch - see HADOOP-12963 -1.patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 58s trunk passed
          +1 compile 6m 9s trunk passed with JDK v1.8.0_74
          +1 compile 6m 48s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 4s trunk passed
          +1 mvnsite 1m 18s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 6s trunk passed
          +1 javadoc 1m 10s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 19s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 57s the patch passed
          +1 compile 5m 40s the patch passed with JDK v1.8.0_74
          +1 javac 5m 40s the patch passed
          +1 compile 6m 39s the patch passed with JDK v1.7.0_95
          +1 javac 6m 39s the patch passed
          +1 checkstyle 1m 22s the patch passed
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 30s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 29s the patch passed
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 18s the patch passed with JDK v1.7.0_95
          -1 unit 6m 33s hadoop-common in the patch failed with JDK v1.8.0_74.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_74.
          -1 unit 6m 54s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 24s Patch generated 2 ASF License warnings.
          65m 4s



          Reason Tests
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker
          JDK v1.7.0_95 Failed junit tests hadoop.net.TestDNS
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795457/HADOOP-12963-1.patch
          JIRA Issue HADOOP-12963
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux e44250f57bea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9a09200
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 58s trunk passed +1 compile 6m 9s trunk passed with JDK v1.8.0_74 +1 compile 6m 48s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 4s trunk passed +1 mvnsite 1m 18s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 6s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 19s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 5m 40s the patch passed with JDK v1.8.0_74 +1 javac 5m 40s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_95 +1 javac 6m 39s the patch passed +1 checkstyle 1m 22s the patch passed +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 29s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 18s the patch passed with JDK v1.7.0_95 -1 unit 6m 33s hadoop-common in the patch failed with JDK v1.8.0_74. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_74. -1 unit 6m 54s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 24s Patch generated 2 ASF License warnings. 65m 4s Reason Tests JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker JDK v1.7.0_95 Failed junit tests hadoop.net.TestDNS JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795457/HADOOP-12963-1.patch JIRA Issue HADOOP-12963 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux e44250f57bea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9a09200 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8931/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          smontgomery Stephen Montgomery added a comment -

          Hi,
          It looks like the build tests core-dumped twice in hadoop-common doing native tests (nothing to do with me, guv! ). Bump up the memory for the jenkins build..? Same error was observed in the build after mine ie https://builds.apache.org/job/PreCommit-HADOOP-Build/8932/.

          I'll let you guys take a look and tell me the next steps. If I don't hear back by Monday, i'll just cancel and resubmit the patch so it'll get another run.

          Thanks,
          Stephen

          Show
          smontgomery Stephen Montgomery added a comment - Hi, It looks like the build tests core-dumped twice in hadoop-common doing native tests (nothing to do with me, guv! ). Bump up the memory for the jenkins build..? Same error was observed in the build after mine ie https://builds.apache.org/job/PreCommit-HADOOP-Build/8932/ . I'll let you guys take a look and tell me the next steps. If I don't hear back by Monday, i'll just cancel and resubmit the patch so it'll get another run. Thanks, Stephen
          Hide
          stevel@apache.org Steve Loughran added a comment -

          don't worry about the core dumps, but

          -describe how you tested it, because jenkins cannot do s3 tests
          submit your patches using the patten {{printf("%s%03d.patch",jira, patch_version). the .patch must come last; the leading 0 numbering to keep sort order. thx

          Show
          stevel@apache.org Steve Loughran added a comment - don't worry about the core dumps, but -describe how you tested it, because jenkins cannot do s3 tests submit your patches using the patten {{printf("%s %03d.patch",jira, patch_version). the .patch must come last; the leading 0 numbering to keep sort order. thx
          Hide
          smontgomery Stephen Montgomery added a comment -

          Updated patch with correct naming structure.

          Show
          smontgomery Stephen Montgomery added a comment - Updated patch with correct naming structure.
          Hide
          smontgomery Stephen Montgomery added a comment -

          Hi Steve,
          Thanks for quick reply. This patch is simply setting a flag on the Amazon S3 Client to use the path style access behaviour by default instead of virtual hosting - see com.amazonaws.services.s3.S3ClientOptions. This is done when the S3AFileSystem initialises the AmazonS3Client. JetS3t has a similar property to do this as well - see s3service.disable-dns-buckets at http://www.jets3t.org/toolkit/configuration.html.

          I submitted a test (org.apache.hadoop.fs.s3a.TestS3AConfiguration.shouldBeAbleToSwitchOnS3PathStyleAccessViaConfigProperty) that simply sets the new Hadoop flag, initialises the new S3AFileSystem and checks that in the newly instantiated AmazonS3Client that it's S3ClientOptions.isPathStyleAccess() is set to true. The S3ClientOptions property interrogation is done via ugly reflection as the property is not retrievable via Amazon S3 SDK.

          When the test runs against "live" S3A buckets, and the path style access switched on, the buckets have be created in the same region as the AmazonS3Client (with default s3.amazonaws.com endpoint specified) otherwise a 301 error thrown (see http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html) which is in the test as well.

          I have patched a running cluster and submitted client jobs using the new flag and it works as expected - it removed the need to have all the virtual hosted buckets specified in the /etc/hosts file. I have also done manual tests specifying the region.amazonaws.com as custom S3A endpoint to bypass the 301 error when I have buckets in different regions. I also used an IPv4 address as the custom S3A endpoint that is a known workaround to switch on path style access in the AmazonS3Client code itself.

          I could have written a few more tests, maybe creating new buckets on the fly in different regions to test for the 301 error but I don't know if this error code is specific to AWS S3 only (and never going to change). The actual AWS S3A operations behaviour doesn't vary when virtual hosting or path style access used. But the upshot is that I'm just setting a flag on the AmazonS3Client instance creation and that small 3 liner probably (!?!) doesn't warrant 100 lines or so of junit code. If you think it does though, I'll go ahead and do it...

          Thanks,
          Stephen

          Show
          smontgomery Stephen Montgomery added a comment - Hi Steve, Thanks for quick reply. This patch is simply setting a flag on the Amazon S3 Client to use the path style access behaviour by default instead of virtual hosting - see com.amazonaws.services.s3.S3ClientOptions. This is done when the S3AFileSystem initialises the AmazonS3Client. JetS3t has a similar property to do this as well - see s3service.disable-dns-buckets at http://www.jets3t.org/toolkit/configuration.html . I submitted a test (org.apache.hadoop.fs.s3a.TestS3AConfiguration.shouldBeAbleToSwitchOnS3PathStyleAccessViaConfigProperty) that simply sets the new Hadoop flag, initialises the new S3AFileSystem and checks that in the newly instantiated AmazonS3Client that it's S3ClientOptions.isPathStyleAccess() is set to true. The S3ClientOptions property interrogation is done via ugly reflection as the property is not retrievable via Amazon S3 SDK. When the test runs against "live" S3A buckets, and the path style access switched on, the buckets have be created in the same region as the AmazonS3Client (with default s3.amazonaws.com endpoint specified) otherwise a 301 error thrown (see http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html ) which is in the test as well. I have patched a running cluster and submitted client jobs using the new flag and it works as expected - it removed the need to have all the virtual hosted buckets specified in the /etc/hosts file. I have also done manual tests specifying the region.amazonaws.com as custom S3A endpoint to bypass the 301 error when I have buckets in different regions. I also used an IPv4 address as the custom S3A endpoint that is a known workaround to switch on path style access in the AmazonS3Client code itself. I could have written a few more tests, maybe creating new buckets on the fly in different regions to test for the 301 error but I don't know if this error code is specific to AWS S3 only (and never going to change). The actual AWS S3A operations behaviour doesn't vary when virtual hosting or path style access used. But the upshot is that I'm just setting a flag on the AmazonS3Client instance creation and that small 3 liner probably (!?!) doesn't warrant 100 lines or so of junit code. If you think it does though, I'll go ahead and do it... Thanks, Stephen
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          -1 mvninstall 10m 19s root in trunk failed.
          -1 compile 10m 58s root in trunk failed with JDK v1.8.0_74.
          +1 compile 10m 37s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 28s trunk passed
          +1 mvnsite 1m 48s trunk passed
          +1 mvneclipse 0m 32s trunk passed
          +1 findbugs 2m 41s trunk passed
          +1 javadoc 1m 33s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 43s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 11m 5s the patch passed with JDK v1.8.0_74
          -1 javac 12m 43s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 93 new + 648 unchanged - 91 fixed = 741 total (was 739)
          +1 javac 11m 5s the patch passed
          +1 compile 10m 11s the patch passed with JDK v1.7.0_95
          +1 javac 10m 11s the patch passed
          +1 checkstyle 1m 32s the patch passed
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 3m 43s the patch passed
          +1 javadoc 1m 49s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95
          -1 unit 10m 39s hadoop-common in the patch failed with JDK v1.8.0_74.
          +1 unit 0m 15s hadoop-aws in the patch passed with JDK v1.8.0_74.
          -1 unit 8m 59s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 23s Patch generated 1 ASF License warnings.
          99m 8s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795528/HADOOP-12963-001.patch
          JIRA Issue HADOOP-12963
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux cef34e9ef995 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a337ceb
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/branch-mvninstall-root.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/branch-compile-root-jdk1.8.0_74.txt
          findbugs v3.0.0
          javac root-jdk1.8.0_74: https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch -1 mvninstall 10m 19s root in trunk failed. -1 compile 10m 58s root in trunk failed with JDK v1.8.0_74. +1 compile 10m 37s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 28s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 0m 32s trunk passed +1 findbugs 2m 41s trunk passed +1 javadoc 1m 33s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 43s trunk passed with JDK v1.7.0_95 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 11m 5s the patch passed with JDK v1.8.0_74 -1 javac 12m 43s root-jdk1.8.0_74 with JDK v1.8.0_74 generated 93 new + 648 unchanged - 91 fixed = 741 total (was 739) +1 javac 11m 5s the patch passed +1 compile 10m 11s the patch passed with JDK v1.7.0_95 +1 javac 10m 11s the patch passed +1 checkstyle 1m 32s the patch passed +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 43s the patch passed +1 javadoc 1m 49s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95 -1 unit 10m 39s hadoop-common in the patch failed with JDK v1.8.0_74. +1 unit 0m 15s hadoop-aws in the patch passed with JDK v1.8.0_74. -1 unit 8m 59s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 23s Patch generated 1 ASF License warnings. 99m 8s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.ipc.TestRPCWaitForProxy JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795528/HADOOP-12963-001.patch JIRA Issue HADOOP-12963 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux cef34e9ef995 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a337ceb Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/branch-compile-root-jdk1.8.0_74.txt findbugs v3.0.0 javac root-jdk1.8.0_74: https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8936/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Why does the test here play with reflection? Why not actually have a test which takes a config entry pointing to a path entry, and use it to read something?

          also in those tests, assertTrues should come with some text explaining why things failed .

          Show
          stevel@apache.org Steve Loughran added a comment - Why does the test here play with reflection? Why not actually have a test which takes a config entry pointing to a path entry, and use it to read something? also in those tests, assertTrues should come with some text explaining why things failed .
          Hide
          smontgomery Stephen Montgomery added a comment -

          Hi,
          1. All I'm/Hadoop doing is setting a switch on the S3 SDK client itself. I'd be testing SDK functionality in Hadoop tests - it's already been tested (and better) by Amazon.
          2. Reflection is used because the S3ClientOptions (and other methods/objects) in the SDK are inaccessible.
          3. There is no behavioural difference between path style and virtual hosting so having a test that retrieves something doesn't really tell me much. Again it's been better tested elsewhere.
          4. Fair point about the assertTrue's

          Feel free to go ahead and modify/delete the test as you see fit.

          Show
          smontgomery Stephen Montgomery added a comment - Hi, 1. All I'm/Hadoop doing is setting a switch on the S3 SDK client itself. I'd be testing SDK functionality in Hadoop tests - it's already been tested (and better) by Amazon. 2. Reflection is used because the S3ClientOptions (and other methods/objects) in the SDK are inaccessible. 3. There is no behavioural difference between path style and virtual hosting so having a test that retrieves something doesn't really tell me much. Again it's been better tested elsewhere. 4. Fair point about the assertTrue's Feel free to go ahead and modify/delete the test as you see fit.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          We need those tests so that

          1. We can catch when amazon change things
          2. we can be sure that the feature actually works
          3. we can see what happens when tested against other people's S3 implementations. Most of the recent patches have covered optimisations in such deployments, and compatibility problems. Having an s3a test lets the implementors see whether/how s3a works here.
          Show
          stevel@apache.org Steve Loughran added a comment - We need those tests so that We can catch when amazon change things we can be sure that the feature actually works we can see what happens when tested against other people's S3 implementations. Most of the recent patches have covered optimisations in such deployments, and compatibility problems. Having an s3a test lets the implementors see whether/how s3a works here.
          Hide
          smontgomery Stephen Montgomery added a comment -

          Updated test to do file lifecycle with path style access switched on.

          Show
          smontgomery Stephen Montgomery added a comment - Updated test to do file lifecycle with path style access switched on.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 18m 39s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 51s trunk passed
          +1 compile 6m 2s trunk passed with JDK v1.8.0_77
          +1 compile 6m 57s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 1m 21s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          +1 findbugs 2m 8s trunk passed
          +1 javadoc 1m 7s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 23s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 0m 59s the patch passed
          +1 compile 6m 9s the patch passed with JDK v1.8.0_77
          +1 javac 6m 9s the patch passed
          +1 compile 6m 58s the patch passed with JDK v1.7.0_95
          +1 javac 6m 58s the patch passed
          +1 checkstyle 1m 6s the patch passed
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 32s the patch passed
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 19s the patch passed with JDK v1.7.0_95
          -1 unit 21m 32s hadoop-common in the patch failed with JDK v1.8.0_77.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_77.
          +1 unit 7m 40s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          99m 59s



          Reason Tests
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797033/HADOOP-12963-002.patch
          JIRA Issue HADOOP-12963
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 9fb0c17a887f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 776b549
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9026/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9026/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9026/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9026/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 18m 39s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 6m 2s trunk passed with JDK v1.8.0_77 +1 compile 6m 57s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 9s trunk passed +1 mvnsite 1m 21s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 2m 8s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 23s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 6m 9s the patch passed with JDK v1.8.0_77 +1 javac 6m 9s the patch passed +1 compile 6m 58s the patch passed with JDK v1.7.0_95 +1 javac 6m 58s the patch passed +1 checkstyle 1m 6s the patch passed +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 32s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 19s the patch passed with JDK v1.7.0_95 -1 unit 21m 32s hadoop-common in the patch failed with JDK v1.8.0_77. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_77. +1 unit 7m 40s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 99m 59s Reason Tests JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797033/HADOOP-12963-002.patch JIRA Issue HADOOP-12963 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 9fb0c17a887f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 776b549 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9026/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9026/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9026/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9026/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1 committed.

          new tests passed. Some other S3A tests were unhappy today against trunk, but it seems unrelated. Anyone who can: please do a test run of the hadoop-aws module today and verify it works for them

          Show
          stevel@apache.org Steve Loughran added a comment - +1 committed. new tests passed. Some other S3A tests were unhappy today against trunk, but it seems unrelated. Anyone who can: please do a test run of the hadoop-aws module today and verify it works for them
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9610 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9610/)
          HADOOP-12963 Allow using path style addressing for accessing the s3 (stevel: rev df18b6e9849c53c51a3d317f1254298edd8b17d1)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9610 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9610/ ) HADOOP-12963 Allow using path style addressing for accessing the s3 (stevel: rev df18b6e9849c53c51a3d317f1254298edd8b17d1) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Hide
          smontgomery Stephen Montgomery added a comment -

          Great stuff. Thanks Steve!

          Show
          smontgomery Stephen Montgomery added a comment - Great stuff. Thanks Steve!
          Hide
          smontgomery Stephen Montgomery added a comment -

          Hi Steve,
          I was wondering if this update has been pigeon-holed for a release yet? We've actually patched a release for a customer so both of us are keeping an eye on this ticket status...!

          Thanks,
          Stephen

          Show
          smontgomery Stephen Montgomery added a comment - Hi Steve, I was wondering if this update has been pigeon-holed for a release yet? We've actually patched a release for a customer so both of us are keeping an eye on this ticket status...! Thanks, Stephen
          Hide
          stevel@apache.org Steve Loughran added a comment -

          It's in 2.8; I missed out setting the version marker there. Next Hadoop release there is.

          you can help by testing that branch and looking at the new stuff coming in the S3A phase II JIRA, reviewing and testing on your local systems...

          Show
          stevel@apache.org Steve Loughran added a comment - It's in 2.8; I missed out setting the version marker there. Next Hadoop release there is. you can help by testing that branch and looking at the new stuff coming in the S3A phase II JIRA, reviewing and testing on your local systems...
          Hide
          smontgomery Stephen Montgomery added a comment -

          Brilliant thanks.

          Show
          smontgomery Stephen Montgomery added a comment - Brilliant thanks.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the contribution folks.
          Steve Loughran I saw the fix version field was empty, and this seems to be in trunk and 2.8, so I updated as such. Please feel free to change.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the contribution folks. Steve Loughran I saw the fix version field was empty, and this seems to be in trunk and 2.8, so I updated as such. Please feel free to change.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Xiao Chen for taking care of this. Removed 3.0.0-alpha1 from fix version/s.

          https://wiki.apache.org/hadoop/HowToCommit

          Always set the "Fix Version" at this point, but please only set a single fix version, the earliest release in which the change will appear.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Xiao Chen for taking care of this. Removed 3.0.0-alpha1 from fix version/s. https://wiki.apache.org/hadoop/HowToCommit Always set the "Fix Version" at this point, but please only set a single fix version, the earliest release in which the change will appear.
          Hide
          xiaochen Xiao Chen added a comment -

          Aha, thanks Akira Ajisaka for correcting me!

          Show
          xiaochen Xiao Chen added a comment - Aha, thanks Akira Ajisaka for correcting me!

            People

            • Assignee:
              smontgomery Stephen Montgomery
              Reporter:
              abaptist Andrew Baptist
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development