Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.9.0, 3.0.0-beta1
    • Fix Version/s: 3.0.0-beta1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      * The s3n:// client has been removed. Please upgrade to the s3a:// client.
      * The s3a's original output stream has been removed, the "fast" output stream is the sole option available. There is no need to explicitly enable this, and trying to disable it (fs.s3a.fast.upload=false) will have no effect.
      Show
      * The s3n:// client has been removed. Please upgrade to the s3a:// client. * The s3a's original output stream has been removed, the "fast" output stream is the sole option available. There is no need to explicitly enable this, and trying to disable it (fs.s3a.fast.upload=false) will have no effect.

      Description

      We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf since Hadoop 2.8

      It's now time to kill S3N off, remove the source, the tests, the transitive dependencies. This patch does that.

      It also removes the obsolete, original s3a output stream; the fast/block upload stream has been stable and is much more manageable and maintained (put differently: we don't ever look at the original S3A output stream, and tell people not to use it for performance reasons).

      As well as cutting the features, this patch updates the aws docs with

      • split out s3n migration page
      • split out troubleshooting page
      • rework of the "uploading data with s3a" section of index.md, as there's no need to discuss the slow upload except in the past tense...all that is needed is to list the buffering and thread tuning options of the block uploader.
      1. HADOOP-14738-002.patch
        113 kB
        Steve Loughran
      2. HADOOP-14738-003.patch
        254 kB
        Steve Loughran
      3. HADOOP-14738-004.patch
        317 kB
        Steve Loughran
      4. HADOOP-14738-005.patch
        316 kB
        Steve Loughran
      5. HADOOP-14738-006.patch
        316 kB
        Aaron Fabbri
      6. HADOOP-14739-001.patch
        111 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Some ideas

          • For those people who explicitly go fs.s3n.impl = org.apache.hadoop.fs.s3native.NativeS3FileSystem, we could retain an FS impl there which extended S3A and warned users off it
          • Could we pick up the old key names (or at least fs.s3a equivalents) as deprecated values? I'm reluctant to do this, as Configuration.getPassword() doesn't handle deprecation.
          Show
          stevel@apache.org Steve Loughran added a comment - Some ideas For those people who explicitly go fs.s3n.impl = org.apache.hadoop.fs.s3native.NativeS3FileSystem , we could retain an FS impl there which extended S3A and warned users off it Could we pick up the old key names (or at least fs.s3a equivalents) as deprecated values? I'm reluctant to do this, as Configuration.getPassword() doesn't handle deprecation.
          Hide
          liuml07 Mingliang Liu added a comment -

          I'm with the proposal.

          For those people who explicitly go fs.s3n.impl = org.apache.hadoop.fs.s3native.NativeS3FileSystem, we could retain an FS impl there which extended S3A and warned users off it

          This is good idea.

          Show
          liuml07 Mingliang Liu added a comment - I'm with the proposal. For those people who explicitly go fs.s3n.impl = org.apache.hadoop.fs.s3native.NativeS3FileSystem, we could retain an FS impl there which extended S3A and warned users off it This is good idea.
          Hide
          aw Allen Wittenauer added a comment -

          Why not just remove it in 3.0?

          Show
          aw Allen Wittenauer added a comment - Why not just remove it in 3.0?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Good Q.

          On one hand: we've not given any warning

          On the other: we have a full replacement, so migration is all you need to do. It's not like we cut S3N out.

          In that case we could go

          1. branch-2, 2.8 -> warn
          2. trunk -> add that wrapper class which tells off user, maybe links to wiki entry on how to migrate. Given changes in management of auth details (which is much more than just changed key names), I'd not want to rush to copying over the old secret names, instead help people move
          Show
          stevel@apache.org Steve Loughran added a comment - Good Q. On one hand: we've not given any warning On the other: we have a full replacement, so migration is all you need to do. It's not like we cut S3N out. In that case we could go branch-2, 2.8 -> warn trunk -> add that wrapper class which tells off user, maybe links to wiki entry on how to migrate. Given changes in management of auth details (which is much more than just changed key names), I'd not want to rush to copying over the old secret names, instead help people move
          Hide
          fabbri Aaron Fabbri added a comment -

          I'd vote to remove s3n in 3.0.. and make this JIRA a blocker to make sure we get this in.

          Show
          fabbri Aaron Fabbri added a comment - I'd vote to remove s3n in 3.0.. and make this JIRA a blocker to make sure we get this in.
          Hide
          andrew.wang Andrew Wang added a comment -

          Is anyone planning to pick this up and complete it by mid-September for beta1?

          Show
          andrew.wang Andrew Wang added a comment - Is anyone planning to pick this up and complete it by mid-September for beta1?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          depends what we think is needed. I'm thinking of: rm docs, test, existing code. Add new section "migrating to s3a" (wiki?) and have the s3n FS impl print this to tell people what to do.

          We can't do an automated migration as the key settings are all different

          Show
          stevel@apache.org Steve Loughran added a comment - depends what we think is needed. I'm thinking of: rm docs, test, existing code. Add new section "migrating to s3a" (wiki?) and have the s3n FS impl print this to tell people what to do. We can't do an automated migration as the key settings are all different
          Hide
          stevel@apache.org Steve Loughran added a comment -

          FWIW. I don't think removal is blocker. Marking as deprecated, yes, and straightforward

          Show
          stevel@apache.org Steve Loughran added a comment - FWIW. I don't think removal is blocker. Marking as deprecated, yes, and straightforward
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks Steve. A strict interpretation of the compat guidelines would say deprecate in 3.0 and remove in 4.0. Is this what we're planning?

          I could see an argument for removal without a deprecation cycle if S3N is a high maintenance burden, we have high confidence that no one uses it, and the same data remains accessible via S3A.

          Show
          andrew.wang Andrew Wang added a comment - Thanks Steve. A strict interpretation of the compat guidelines would say deprecate in 3.0 and remove in 4.0. Is this what we're planning? I could see an argument for removal without a deprecation cycle if S3N is a high maintenance burden, we have high confidence that no one uses it, and the same data remains accessible via S3A.
          Hide
          fabbri Aaron Fabbri added a comment -

          I'm happy to put up a patch for this once we have consensus.

          I could see an argument for removal without a deprecation cycle if S3N is a high maintenance burden, we have high confidence that no one uses it, and the same data remains accessible via S3A.

          I think #1 and #3 are true, but don't have confidence that no one uses it.

          Show
          fabbri Aaron Fabbri added a comment - I'm happy to put up a patch for this once we have consensus. I could see an argument for removal without a deprecation cycle if S3N is a high maintenance burden, we have high confidence that no one uses it, and the same data remains accessible via S3A. I think #1 and #3 are true, but don't have confidence that no one uses it.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm doing this with the doc updates of HADOOP-14826m as they all go together, and marking jets3t.jar as provided, so it doesn't go downstream any more, targeting 3.0

          FWIW, I can see nobody has been running the s3n tests for a while, as the new tests from HADOOP-14397 are failing. I'll handle that here too

          testCreateFileOverExistingFileNoOverwrite(org.apache.hadoop.fs.contract.s3n.ITestS3NContractCreate)  Time elapsed: 1.292 sec  <<< ERROR!
          java.io.IOException: createNonRecursive unsupported for this filesystem class org.apache.hadoop.fs.s3native.NativeS3FileSystem
          	at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1304)
          	at org.apache.hadoop.fs.FileSystem$FileSystemDataOutputStreamBuilder.build(FileSystem.java:4163)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:179)
          	at org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreateFileOverExistingFileNoOverwrite(AbstractContractCreateTest.java:79)
          	at org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreateFileOverExistingFileNoOverwrite(AbstractContractCreateTest.java:95)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Show
          stevel@apache.org Steve Loughran added a comment - I'm doing this with the doc updates of HADOOP-14826 m as they all go together, and marking jets3t.jar as provided, so it doesn't go downstream any more, targeting 3.0 FWIW, I can see nobody has been running the s3n tests for a while, as the new tests from HADOOP-14397 are failing. I'll handle that here too testCreateFileOverExistingFileNoOverwrite(org.apache.hadoop.fs.contract.s3n.ITestS3NContractCreate) Time elapsed: 1.292 sec <<< ERROR! java.io.IOException: createNonRecursive unsupported for this filesystem class org.apache.hadoop.fs.s3native.NativeS3FileSystem at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1304) at org.apache.hadoop.fs.FileSystem$FileSystemDataOutputStreamBuilder.build(FileSystem.java:4163) at org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:179) at org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreateFileOverExistingFileNoOverwrite(AbstractContractCreateTest.java:79) at org.apache.hadoop.fs.contract.AbstractContractCreateTest.testCreateFileOverExistingFileNoOverwrite(AbstractContractCreateTest.java:95) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 001

          1. tags everyhting as @ deprecated
          2. logs a warning on the first instance inited in a VM
          3. Implement createNonRecursive so that ITestS3NContractCreate works again.
          4. removed jets3t as an exported depencency of hadoop-aws
          5. moved all docs over to an s3n.md file
          6. provide instructions on migrating
          7. main docs are s3a only now, so clean up text, titles, anchors
          8. review docs, add anchors, promote headings
          9. move all troubleshooting into troubleshooting.md
          10. Qualify consistency warkings with more refs to using s3guard
          11. Add a few external links (EMR fs docs, HDP on distcp to s3, which is something I co-authored a while back. We could copy that in directly, but copy-and-paste docs is as much a maintenance issue as code copy and paste.

          See github for the MD to be rendered.

          The jets3t export is something to debate It's a half-way migrate: you can still use s3n, you just need to sort out the classpath. And it keeps the maven builds downstream much cleaner.

          Tested against s3 ireland. I even found the regression in ITestS3NContractCreate which we'd not picked up before (I fixed, rather than skip the tests because the fix was a simple copy & paste from S3a)

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 001 tags everyhting as @ deprecated logs a warning on the first instance inited in a VM Implement createNonRecursive so that ITestS3NContractCreate works again. removed jets3t as an exported depencency of hadoop-aws moved all docs over to an s3n.md file provide instructions on migrating main docs are s3a only now, so clean up text, titles, anchors review docs, add anchors, promote headings move all troubleshooting into troubleshooting.md Qualify consistency warkings with more refs to using s3guard Add a few external links (EMR fs docs, HDP on distcp to s3, which is something I co-authored a while back. We could copy that in directly, but copy-and-paste docs is as much a maintenance issue as code copy and paste. See github for the MD to be rendered. The jets3t export is something to debate It's a half-way migrate: you can still use s3n, you just need to sort out the classpath. And it keeps the maven builds downstream much cleaner. Tested against s3 ireland. I even found the regression in ITestS3NContractCreate which we'd not picked up before (I fixed, rather than skip the tests because the fix was a simple copy & paste from S3a)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          FWIW, I'm hoping to cut completely from 3.1 because migration is so straightforward: change URLs, change auth keys.

          It's not like a feature "s3 connectivity" is gone, just some of the config/use parameters

          Show
          stevel@apache.org Steve Loughran added a comment - FWIW, I'm hoping to cut completely from 3.1 because migration is so straightforward: change URLs, change auth keys. It's not like a feature "s3 connectivity" is gone, just some of the config/use parameters
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 14m 57s trunk passed
          +1 compile 0m 23s trunk passed
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 findbugs 0m 34s trunk passed
          +1 javadoc 0m 16s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 20s the patch passed
          -1 javac 0m 20s hadoop-tools_hadoop-aws generated 71 new + 6 unchanged - 1 fixed = 77 total (was 7)
          +1 checkstyle 0m 14s the patch passed
          -1 mvnsite 0m 22s hadoop-aws in the patch failed.
          -1 whitespace 0m 0s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 0m 37s the patch passed
          +1 javadoc 0m 12s the patch passed
                Other Tests
          +1 unit 0m 40s hadoop-aws in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          21m 34s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HADOOP-14738
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885275/HADOOP-14739-001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 72e19f0d2525 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef87d34
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-aws.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-aws.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 57s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 28s trunk passed +1 findbugs 0m 34s trunk passed +1 javadoc 0m 16s trunk passed       Patch Compile Tests +1 mvninstall 0m 21s the patch passed +1 compile 0m 20s the patch passed -1 javac 0m 20s hadoop-tools_hadoop-aws generated 71 new + 6 unchanged - 1 fixed = 77 total (was 7) +1 checkstyle 0m 14s the patch passed -1 mvnsite 0m 22s hadoop-aws in the patch failed. -1 whitespace 0m 0s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 0m 37s the patch passed +1 javadoc 0m 12s the patch passed       Other Tests +1 unit 0m 40s hadoop-aws in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 21m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HADOOP-14738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885275/HADOOP-14739-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 72e19f0d2525 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef87d34 Default Java 1.8.0_144 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-aws.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-aws.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13161/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          All the javac warnings are related to the fact that I've tagged the NativeS3 classes as @Deprecated. That's part of the patch.

          Show
          stevel@apache.org Steve Loughran added a comment - All the javac warnings are related to the fact that I've tagged the NativeS3 classes as @Deprecated. That's part of the patch.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          see HADOOP-14332: add some text on effective SSE. Maybe: move encryption out to its own doc too?

          Show
          stevel@apache.org Steve Loughran added a comment - see HADOOP-14332 : add some text on effective SSE. Maybe: move encryption out to its own doc too?
          Hide
          fabbri Aaron Fabbri added a comment -

          This looks great. +1 (non-binding) The docs are really coming together, thanks for the improvements.

          Show
          fabbri Aaron Fabbri added a comment - This looks great. +1 (non-binding) The docs are really coming together, thanks for the improvements.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Steve, thanks for the patch,

          I'd rather we keep the jets3t scope at "compile" if we're going to keep it at all. It punts the classpath problems to end users, who are less equipped to deal with it. IMO features should be fully supported or not supported at all.

          Removing in 3.0 also seems better than removing in 3.1 since at least 3.0 is a major release.

          Whether to remove or not depends on how often S3N is used, and how hard it is to use S3A instead. I'll defer to subject matter experts like yourself and Aaron on this question. If it's not used and easy to migrate, then I'm +1 on removing in 3.0.

          Show
          andrew.wang Andrew Wang added a comment - Hi Steve, thanks for the patch, I'd rather we keep the jets3t scope at "compile" if we're going to keep it at all. It punts the classpath problems to end users, who are less equipped to deal with it. IMO features should be fully supported or not supported at all. Removing in 3.0 also seems better than removing in 3.1 since at least 3.0 is a major release. Whether to remove or not depends on how often S3N is used, and how hard it is to use S3A instead. I'll defer to subject matter experts like yourself and Aaron on this question. If it's not used and easy to migrate, then I'm +1 on removing in 3.0.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          A full cut of s3n? I'm happy with that.

          in which case I'd just have a stub child FS which declared itself as missing & switch the s3 docs to a "how to migrate" doc alone

          Show
          stevel@apache.org Steve Loughran added a comment - A full cut of s3n? I'm happy with that. in which case I'd just have a stub child FS which declared itself as missing & switch the s3 docs to a "how to migrate" doc alone
          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-14738 002: reinstate jets3t; add index.vm ref, and more anchors in the aws sections

          This patch still has the s3n binaries & tests, so can be applied to branch-2 . I'm going to do another iteration with s3n pulled other than a stub class; tests will be cut along with config details

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-14738 002: reinstate jets3t; add index.vm ref, and more anchors in the aws sections This patch still has the s3n binaries & tests, so can be applied to branch-2 . I'm going to do another iteration with s3n pulled other than a stub class; tests will be cut along with config details
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 003

          S3N

          • remove all traces of s3n throughout the project, XML, docs, resources, test-resources
          • NativeS3FileSystem now a stub class which throws an IOE in its initialize call.
          • simplified other bits of docs (e.g. testing.md) where there's now less to worry about
          • Jets3t from all poms &c

          docs

          • add root index.md.vm entry as requested
          • fix site generation (DOXIA-533) by removing all level4 headers
          • encryption doc added; mix of HDP content and some other bits
          • more subsections in troubleshooting
          • review all docs, site generation, ...
          Show
          stevel@apache.org Steve Loughran added a comment - Patch 003 S3N remove all traces of s3n throughout the project, XML, docs, resources, test-resources NativeS3FileSystem now a stub class which throws an IOE in its initialize call. simplified other bits of docs (e.g. testing.md) where there's now less to worry about Jets3t from all poms &c docs add root index.md.vm entry as requested fix site generation ( DOXIA-533 ) by removing all level4 headers encryption doc added; mix of HDP content and some other bits more subsections in troubleshooting review all docs, site generation, ...
          Hide
          stevel@apache.org Steve Loughran added a comment -

          this patch is targeting 3.0 only, BTW. We could have 2.9 do the earlier warning-of-pending-doom log option & leave the docs alone there

          Show
          stevel@apache.org Steve Loughran added a comment - this patch is targeting 3.0 only, BTW. We could have 2.9 do the earlier warning-of-pending-doom log option & leave the docs alone there
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 23 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 54s Maven dependency ordering for branch
          +1 mvninstall 15m 58s trunk passed
          +1 compile 17m 38s trunk passed
          +1 checkstyle 2m 16s trunk passed
          +1 mvnsite 2m 35s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster
          +1 findbugs 2m 8s trunk passed
          +1 javadoc 1m 57s trunk passed
                Patch Compile Tests
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 4m 8s the patch passed
          +1 compile 13m 43s the patch passed
          +1 javac 13m 43s root generated 0 new + 1282 unchanged - 1 fixed = 1282 total (was 1283)
          -0 checkstyle 2m 6s root: The patch generated 2 new + 23 unchanged - 103 fixed = 25 total (was 126)
          +1 mvnsite 2m 31s the patch passed
          -1 whitespace 0m 0s The patch has 30 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 8s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster
          -1 findbugs 0m 58s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 2m 19s the patch passed
                Other Tests
          +1 unit 0m 21s hadoop-project in the patch passed.
          +1 unit 8m 18s hadoop-common in the patch passed.
          +1 unit 0m 56s hadoop-aws in the patch passed.
          +1 unit 0m 29s hadoop-client-minicluster in the patch passed.
          +1 asflicense 0m 44s The patch does not generate ASF License warnings.
          105m 18s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-aws
            Comparison of String objects using == or != in org.apache.hadoop.fs.s3native.S3xLoginHelper.checkPath(Configuration, URI, Path, int) At S3xLoginHelper.java:== or != in org.apache.hadoop.fs.s3native.S3xLoginHelper.checkPath(Configuration, URI, Path, int) At S3xLoginHelper.java:[line 228]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HADOOP-14738
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885623/HADOOP-14738-003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 0bb48f236852 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1f3bc63
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-client-modules/hadoop-client-minicluster U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 23 new or modified test files.       trunk Compile Tests 0 mvndep 0m 54s Maven dependency ordering for branch +1 mvninstall 15m 58s trunk passed +1 compile 17m 38s trunk passed +1 checkstyle 2m 16s trunk passed +1 mvnsite 2m 35s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster +1 findbugs 2m 8s trunk passed +1 javadoc 1m 57s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 4m 8s the patch passed +1 compile 13m 43s the patch passed +1 javac 13m 43s root generated 0 new + 1282 unchanged - 1 fixed = 1282 total (was 1283) -0 checkstyle 2m 6s root: The patch generated 2 new + 23 unchanged - 103 fixed = 25 total (was 126) +1 mvnsite 2m 31s the patch passed -1 whitespace 0m 0s The patch has 30 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 8s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster -1 findbugs 0m 58s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 19s the patch passed       Other Tests +1 unit 0m 21s hadoop-project in the patch passed. +1 unit 8m 18s hadoop-common in the patch passed. +1 unit 0m 56s hadoop-aws in the patch passed. +1 unit 0m 29s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 105m 18s Reason Tests FindBugs module:hadoop-tools/hadoop-aws   Comparison of String objects using == or != in org.apache.hadoop.fs.s3native.S3xLoginHelper.checkPath(Configuration, URI, Path, int) At S3xLoginHelper.java:== or != in org.apache.hadoop.fs.s3native.S3xLoginHelper.checkPath(Configuration, URI, Path, int) At S3xLoginHelper.java: [line 228] Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HADOOP-14738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885623/HADOOP-14738-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 0bb48f236852 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1f3bc63 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-client-modules/hadoop-client-minicluster U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13179/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          while doing the cleanup, cut the original output stream from the source, tests, and all the docs

          Show
          stevel@apache.org Steve Loughran added a comment - while doing the cleanup, cut the original output stream from the source, tests, and all the docs
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          Patch 004; also remove the old S3AOutputStream,

          Cutting the classic S3A output stream removes another scale test, implicitly increases test coverage, and simplifies the doc. I've retained a check for the option in the S3A initialize() method, but all it does it warn the user that the option is gone.

          Docs all reviewed. cross-doc links reviewed, etc.

          Tested: s3 ireland, with and without s3guard; at scale

          Show
          stevel@apache.org Steve Loughran added a comment - - edited Patch 004; also remove the old S3AOutputStream, Cutting the classic S3A output stream removes another scale test, implicitly increases test coverage, and simplifies the doc. I've retained a check for the option in the S3A initialize() method, but all it does it warn the user that the option is gone. Docs all reviewed. cross-doc links reviewed, etc. Tested: s3 ireland, with and without s3guard; at scale
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 30 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 15m 30s trunk passed
          +1 compile 15m 24s trunk passed
          +1 checkstyle 2m 2s trunk passed
          +1 mvnsite 2m 22s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster
          +1 findbugs 2m 14s trunk passed
          +1 javadoc 1m 50s trunk passed
                Patch Compile Tests
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 4m 14s the patch passed
          +1 compile 12m 21s the patch passed
          -1 javac 12m 21s root generated 1 new + 1282 unchanged - 1 fixed = 1283 total (was 1283)
          -0 checkstyle 2m 8s root: The patch generated 2 new + 34 unchanged - 104 fixed = 36 total (was 138)
          +1 mvnsite 2m 26s the patch passed
          -1 whitespace 0m 0s The patch has 32 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 8s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster
          +1 findbugs 2m 44s the patch passed
          +1 javadoc 2m 4s the patch passed
                Other Tests
          +1 unit 0m 21s hadoop-project in the patch passed.
          -1 unit 8m 38s hadoop-common in the patch failed.
          +1 unit 0m 50s hadoop-aws in the patch passed.
          +1 unit 0m 27s hadoop-client-minicluster in the patch passed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          99m 12s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HADOOP-14738
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885817/HADOOP-14738-004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 30e751476ddf 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2adf8be
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-client-modules/hadoop-client-minicluster U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 30 new or modified test files.       trunk Compile Tests 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 15m 30s trunk passed +1 compile 15m 24s trunk passed +1 checkstyle 2m 2s trunk passed +1 mvnsite 2m 22s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster +1 findbugs 2m 14s trunk passed +1 javadoc 1m 50s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 4m 14s the patch passed +1 compile 12m 21s the patch passed -1 javac 12m 21s root generated 1 new + 1282 unchanged - 1 fixed = 1283 total (was 1283) -0 checkstyle 2m 8s root: The patch generated 2 new + 34 unchanged - 104 fixed = 36 total (was 138) +1 mvnsite 2m 26s the patch passed -1 whitespace 0m 0s The patch has 32 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 8s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster +1 findbugs 2m 44s the patch passed +1 javadoc 2m 4s the patch passed       Other Tests +1 unit 0m 21s hadoop-project in the patch passed. -1 unit 8m 38s hadoop-common in the patch failed. +1 unit 0m 50s hadoop-aws in the patch passed. +1 unit 0m 27s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 99m 12s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HADOOP-14738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885817/HADOOP-14738-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 30e751476ddf 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2adf8be Default Java 1.8.0_144 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-client-modules/hadoop-client-minicluster U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13190/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. new javac is the deprecation of S3AFileSystem
          1. noticed there's a couple of tests which assume that the default output stream is the classic one, with subclasses for the block stream. They can be elided, leaving only the block ones
          Show
          stevel@apache.org Steve Loughran added a comment - new javac is the deprecation of S3AFileSystem noticed there's a couple of tests which assume that the default output stream is the classic one, with subclasses for the block stream. They can be elided, leaving only the block ones
          Hide
          fabbri Aaron Fabbri added a comment -

          Looking at the latest patch now..

          Show
          fabbri Aaron Fabbri added a comment - Looking at the latest patch now..
          Hide
          fabbri Aaron Fabbri added a comment -

          First off, removing a bunch of older code brings me joy. Thanks for that. This looks good overall.

          The stub s3n FS is a nice thing to do for users. I tried it out but just got this:

          $ hadoop fs -ls s3n://fabbri-dev/
          2017-09-07 17:34:32,867 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
          ls: No FileSystem for scheme "s3n"
          

          It does work if I use s3a:// though.

          --- a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties\
          ...
          -log4j.logger.com.amazonaws=ERROR
          -log4j.logger.com.amazonaws.http.AmazonHttpClient=ERROR
          -log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=WARN
          +#log4j.logger.com.amazonaws=ERROR
          +#log4j.logger.com.amazonaws.http.AmazonHttpClient=ERROR
          +#log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=WARN
          

          Does this change defaults to INFO?

          +S3N wasa Hadoop filesystem client which can read or write data stored
          +in Amazon S3. It uses URLs with the schema `s3n://`.
          

          I like the migration document. Small typo here "wasa".

          +    hdfs fs -ls s3a://my-bucket/
          ...
          +means of verifying that the key and secret can access a bucket;
          +
          +    hdfs fs -ls s3a://key:secret@my-bucket/
          

          Probably existing stuff, but should this be `hadoop fs` or `hdfs dfs`?

          I applied the patch and will go ahead and run all tests in hadoop-aws.

          Show
          fabbri Aaron Fabbri added a comment - First off, removing a bunch of older code brings me joy. Thanks for that. This looks good overall. The stub s3n FS is a nice thing to do for users. I tried it out but just got this: $ hadoop fs -ls s3n://fabbri-dev/ 2017-09-07 17:34:32,867 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ls: No FileSystem for scheme "s3n" It does work if I use s3a:// though. --- a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties\ ... -log4j.logger.com.amazonaws=ERROR -log4j.logger.com.amazonaws.http.AmazonHttpClient=ERROR -log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=WARN +#log4j.logger.com.amazonaws=ERROR +#log4j.logger.com.amazonaws.http.AmazonHttpClient=ERROR +#log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=WARN Does this change defaults to INFO? +S3N wasa Hadoop filesystem client which can read or write data stored +in Amazon S3. It uses URLs with the schema `s3n://`. I like the migration document. Small typo here "wasa". + hdfs fs -ls s3a://my-bucket/ ... +means of verifying that the key and secret can access a bucket; + + hdfs fs -ls s3a://key:secret@my-bucket/ Probably existing stuff, but should this be `hadoop fs` or `hdfs dfs`? I applied the patch and will go ahead and run all tests in hadoop-aws.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'd completely pulled the s3n discovery, but yes, I should reinstate that. will do.

          Info levels: I think the logging had someone got cranked up too high; s3a should be logging at info, shouldn't it? that's the assumption the logs are on.

          Having actually looked at the code, we don't actually log much at info through that specific log. it's all either WARN or DEBUG. Reinstating the diff

          Show
          stevel@apache.org Steve Loughran added a comment - I'd completely pulled the s3n discovery, but yes, I should reinstate that. will do. Info levels: I think the logging had someone got cranked up too high; s3a should be logging at info, shouldn't it? that's the assumption the logs are on. Having actually looked at the code, we don't actually log much at info through that specific log. it's all either WARN or DEBUG. Reinstating the diff
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 005: address Aaron's comments.

          test: s3 ireland

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 005: address Aaron's comments. test: s3 ireland
          Hide
          fabbri Aaron Fabbri added a comment -

          Attaching v6 patch.. Just the v5 patch rebased (and whitespace fixed) on latest trunk (there was a conflict w/ the recent list v2 change in index.md)

          Show
          fabbri Aaron Fabbri added a comment - Attaching v6 patch.. Just the v5 patch rebased (and whitespace fixed) on latest trunk (there was a conflict w/ the recent list v2 change in index.md)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 30 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 41s Maven dependency ordering for branch
          +1 mvninstall 17m 18s trunk passed
          +1 compile 20m 53s trunk passed
          +1 checkstyle 2m 28s trunk passed
          +1 mvnsite 2m 57s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster
          +1 findbugs 2m 39s trunk passed
          +1 javadoc 2m 45s trunk passed
                Patch Compile Tests
          0 mvndep 0m 41s Maven dependency ordering for patch
          -1 mvninstall 3m 10s hadoop-client-minicluster in the patch failed.
          +1 compile 16m 22s the patch passed
          -1 javac 16m 22s root generated 1 new + 1282 unchanged - 1 fixed = 1283 total (was 1283)
          -0 checkstyle 2m 20s root: The patch generated 2 new + 33 unchanged - 105 fixed = 35 total (was 138)
          +1 mvnsite 2m 33s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 7s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster
          +1 findbugs 2m 33s the patch passed
          +1 javadoc 2m 11s the patch passed
                Other Tests
          +1 unit 0m 22s hadoop-project in the patch passed.
          +1 unit 8m 2s hadoop-common in the patch passed.
          +1 unit 0m 51s hadoop-aws in the patch passed.
          +1 unit 0m 30s hadoop-client-minicluster in the patch passed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          117m 54s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HADOOP-14738
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886978/HADOOP-14738-006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 275d8806c7ba 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bb34ae9
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/artifact/patchprocess/patch-mvninstall-hadoop-client-modules_hadoop-client-minicluster.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-client-modules/hadoop-client-minicluster U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 30 new or modified test files.       trunk Compile Tests 0 mvndep 0m 41s Maven dependency ordering for branch +1 mvninstall 17m 18s trunk passed +1 compile 20m 53s trunk passed +1 checkstyle 2m 28s trunk passed +1 mvnsite 2m 57s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster +1 findbugs 2m 39s trunk passed +1 javadoc 2m 45s trunk passed       Patch Compile Tests 0 mvndep 0m 41s Maven dependency ordering for patch -1 mvninstall 3m 10s hadoop-client-minicluster in the patch failed. +1 compile 16m 22s the patch passed -1 javac 16m 22s root generated 1 new + 1282 unchanged - 1 fixed = 1283 total (was 1283) -0 checkstyle 2m 20s root: The patch generated 2 new + 33 unchanged - 105 fixed = 35 total (was 138) +1 mvnsite 2m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 7s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster +1 findbugs 2m 33s the patch passed +1 javadoc 2m 11s the patch passed       Other Tests +1 unit 0m 22s hadoop-project in the patch passed. +1 unit 8m 2s hadoop-common in the patch passed. +1 unit 0m 51s hadoop-aws in the patch passed. +1 unit 0m 30s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 117m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HADOOP-14738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886978/HADOOP-14738-006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 275d8806c7ba 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bb34ae9 Default Java 1.8.0_144 findbugs v3.1.0-RC1 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/artifact/patchprocess/patch-mvninstall-hadoop-client-modules_hadoop-client-minicluster.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-client-modules/hadoop-client-minicluster U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13281/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment - - edited

          javac warning is existing FAST_UPLOAD constant deprecation.

          mvninstall issue w/ minicluster is a dependency version issue, with minicluster using a non-SNAPSHOT build number:

          [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @ hadoop-client-minicluster ---
          [WARNING] 
          Dependency convergence error for org.apache.hadoop:hadoop-annotations:3.1.0-20170913.230947-58 paths to dependency are:
          +-org.apache.hadoop:hadoop-client-minicluster:3.1.0-SNAPSHOT
            +-org.apache.hadoop:hadoop-annotations:3.1.0-20170913.230947-58
          and
          +-org.apache.hadoop:hadoop-client-minicluster:3.1.0-SNAPSHOT
            +-org.apache.hadoop:hadoop-minicluster:3.1.0-SNAPSHOT
              +-org.apache.hadoop:hadoop-yarn-server-tests:3.1.0-SNAPSHOT
                +-org.apache.hadoop:hadoop-annotations:3.1.0-SNAPSHOT
          

          I'm not sure were that 3.0.1-2017... version is coming from?

          Show
          fabbri Aaron Fabbri added a comment - - edited javac warning is existing FAST_UPLOAD constant deprecation. mvninstall issue w/ minicluster is a dependency version issue, with minicluster using a non-SNAPSHOT build number: [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @ hadoop-client-minicluster --- [WARNING] Dependency convergence error for org.apache.hadoop:hadoop-annotations:3.1.0-20170913.230947-58 paths to dependency are: +-org.apache.hadoop:hadoop-client-minicluster:3.1.0-SNAPSHOT +-org.apache.hadoop:hadoop-annotations:3.1.0-20170913.230947-58 and +-org.apache.hadoop:hadoop-client-minicluster:3.1.0-SNAPSHOT +-org.apache.hadoop:hadoop-minicluster:3.1.0-SNAPSHOT +-org.apache.hadoop:hadoop-yarn-server-tests:3.1.0-SNAPSHOT +-org.apache.hadoop:hadoop-annotations:3.1.0-SNAPSHOT I'm not sure were that 3.0.1-2017... version is coming from?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 30 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 17m 9s trunk passed
          +1 compile 18m 51s trunk passed
          +1 checkstyle 2m 22s trunk passed
          +1 mvnsite 2m 33s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster
          +1 findbugs 2m 31s trunk passed
          +1 javadoc 1m 56s trunk passed
                Patch Compile Tests
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 4m 8s the patch passed
          +1 compile 13m 46s the patch passed
          -1 javac 13m 46s root generated 1 new + 1282 unchanged - 1 fixed = 1283 total (was 1283)
          -0 checkstyle 2m 23s root: The patch generated 2 new + 33 unchanged - 105 fixed = 35 total (was 138)
          +1 mvnsite 2m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 8s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster
          +1 findbugs 2m 30s the patch passed
          +1 javadoc 2m 8s the patch passed
                Other Tests
          +1 unit 0m 20s hadoop-project in the patch passed.
          +1 unit 9m 8s hadoop-common in the patch passed.
          +1 unit 1m 1s hadoop-aws in the patch passed.
          +1 unit 0m 29s hadoop-client-minicluster in the patch passed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          111m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HADOOP-14738
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886978/HADOOP-14738-006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 8be6421f56a3 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e0b3c64
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13285/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13285/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13285/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-client-modules/hadoop-client-minicluster U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13285/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 30 new or modified test files.       trunk Compile Tests 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 17m 9s trunk passed +1 compile 18m 51s trunk passed +1 checkstyle 2m 22s trunk passed +1 mvnsite 2m 33s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster +1 findbugs 2m 31s trunk passed +1 javadoc 1m 56s trunk passed       Patch Compile Tests 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 4m 8s the patch passed +1 compile 13m 46s the patch passed -1 javac 13m 46s root generated 1 new + 1282 unchanged - 1 fixed = 1283 total (was 1283) -0 checkstyle 2m 23s root: The patch generated 2 new + 33 unchanged - 105 fixed = 35 total (was 138) +1 mvnsite 2m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 8s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster +1 findbugs 2m 30s the patch passed +1 javadoc 2m 8s the patch passed       Other Tests +1 unit 0m 20s hadoop-project in the patch passed. +1 unit 9m 8s hadoop-common in the patch passed. +1 unit 1m 1s hadoop-aws in the patch passed. +1 unit 0m 29s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 111m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HADOOP-14738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886978/HADOOP-14738-006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 8be6421f56a3 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0b3c64 Default Java 1.8.0_144 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13285/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13285/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13285/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-client-modules/hadoop-client-minicluster U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13285/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          the minicluster thing is probably the VM getting mixed up; don't worry about it & see if it goes away. If it doesn't, it's often a sign of getting version numbers mixed up in poms after someone bumps up the artifact numbers of things.

          I'll do a local check without any snapshot JARs built today & see what happens

          Show
          stevel@apache.org Steve Loughran added a comment - the minicluster thing is probably the VM getting mixed up; don't worry about it & see if it goes away. If it doesn't, it's often a sign of getting version numbers mixed up in poms after someone bumps up the artifact numbers of things. I'll do a local check without any snapshot JARs built today & see what happens
          Hide
          stevel@apache.org Steve Loughran added a comment -

          build works for me

          Show
          stevel@apache.org Steve Loughran added a comment - build works for me
          Hide
          fabbri Aaron Fabbri added a comment -

          Yep I'd relaunched the jenkins run and the minicluster dependency thing went away. Will do an extra round of testing and commit this if everything looks good.

          Show
          fabbri Aaron Fabbri added a comment - Yep I'd relaunched the jenkins run and the minicluster dependency thing went away. Will do an extra round of testing and commit this if everything looks good.
          Hide
          fabbri Aaron Fabbri added a comment -

          Committed to trunk after re-running all hadoop-aws tests in us-west-2.

          Thanks for your work on this Steve Loughran

          Show
          fabbri Aaron Fabbri added a comment - Committed to trunk after re-running all hadoop-aws tests in us-west-2. Thanks for your work on this Steve Loughran
          Hide
          andrew.wang Andrew Wang added a comment -

          Aaron, could you also backport to branch-3.0 for beta1? trunk is on 3.1.0 now.

          Show
          andrew.wang Andrew Wang added a comment - Aaron, could you also backport to branch-3.0 for beta1? trunk is on 3.1.0 now.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12875 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12875/)
          HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. (fabbri: rev 49467165a57fb77932d1d526796624b88ebacd91)

          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/package.html
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • (edit) hadoop-project/src/site/markdown/index.md.vm
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractOpen.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
          • (delete) hadoop-common-project/hadoop-common/src/test/resources/jets3t.properties
          • (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractCreate.java
          • (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/InMemoryNativeFileSystemStore.java
          • (edit) hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractMkdir.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABlockOutputArray.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          • (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigRedactor.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/NativeS3Contract.java
          • (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/S3Credentials.java
          • (edit) hadoop-tools/hadoop-aws/dev-support/findbugs-exclude.xml
          • (edit) hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSECBlockOutputStream.java
          • (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
          • (edit) hadoop-tools/hadoop-aws/pom.xml
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/TestS3NInMemoryFileSystem.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/AbstractSTestS3AHugeFiles.java
          • (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractRootDir.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/ITestJets3tNativeFileSystemStore.java
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          • (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
          • (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeFileSystemStore.java
          • (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/FileMetadata.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/S3NInMemoryFileSystem.java
          • (add) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/TestS3Credentials.java
          • (delete) hadoop-tools/hadoop-aws/src/test/resources/contract/s3n.xml
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSES3BlockOutputStream.java
          • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractDelete.java
          • (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/S3NativeFileSystemConfigKeys.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AHugeFilesClassicOutput.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKeyBlockOutputStream.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractRename.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/ITestJets3tNativeS3FileSystemContract.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/ITestInMemoryNativeS3FileSystemContract.java
          • (add) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3n.md
          • (edit) hadoop-common-project/hadoop-common/src/test/resources/core-site.xml
          • (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
          • (edit) hadoop-project/pom.xml
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCp.java
          • (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractSeek.java
          • (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/S3Exception.java
          • (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/PartialListing.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12875 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12875/ ) HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. (fabbri: rev 49467165a57fb77932d1d526796624b88ebacd91) (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/package.html (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java (edit) hadoop-project/src/site/markdown/index.md.vm (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractOpen.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java (delete) hadoop-common-project/hadoop-common/src/test/resources/jets3t.properties (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractCreate.java (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/InMemoryNativeFileSystemStore.java (edit) hadoop-common-project/hadoop-common/src/main/conf/log4j.properties (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractMkdir.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABlockOutputArray.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigRedactor.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/NativeS3Contract.java (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/S3Credentials.java (edit) hadoop-tools/hadoop-aws/dev-support/findbugs-exclude.xml (edit) hadoop-tools/hadoop-aws/src/test/resources/log4j.properties (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSECBlockOutputStream.java (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md (edit) hadoop-tools/hadoop-aws/pom.xml (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/TestS3NInMemoryFileSystem.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/AbstractSTestS3AHugeFiles.java (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractRootDir.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/ITestJets3tNativeFileSystemStore.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeFileSystemStore.java (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/FileMetadata.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/S3NInMemoryFileSystem.java (add) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/TestS3Credentials.java (delete) hadoop-tools/hadoop-aws/src/test/resources/contract/s3n.xml (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSES3BlockOutputStream.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractDelete.java (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/S3NativeFileSystemConfigKeys.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AHugeFilesClassicOutput.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKeyBlockOutputStream.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/NativeS3FileSystemContractBaseTest.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractRename.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/ITestJets3tNativeS3FileSystemContract.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3native/ITestInMemoryNativeS3FileSystemContract.java (add) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3n.md (edit) hadoop-common-project/hadoop-common/src/test/resources/core-site.xml (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md (edit) hadoop-project/pom.xml (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCp.java (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3n/ITestS3NContractSeek.java (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/S3Exception.java (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/PartialListing.java
          Hide
          fabbri Aaron Fabbri added a comment -

          Thanks Andrew Wang. I'm re-running tests and will commit to branch-3.0 shortly.

          Show
          fabbri Aaron Fabbri added a comment - Thanks Andrew Wang . I'm re-running tests and will commit to branch-3.0 shortly.
          Hide
          fabbri Aaron Fabbri added a comment -

          Committed to branch-3.0 as well.

          Show
          fabbri Aaron Fabbri added a comment - Committed to branch-3.0 as well.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks for committing this. It's always nice to delete things: less maintenance forever. Test runs speed up too

          Show
          stevel@apache.org Steve Loughran added a comment - thanks for committing this. It's always nice to delete things: less maintenance forever. Test runs speed up too

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development