Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.1.0
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      This is a Yetus only JIRA created to have Yetus review the HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR https://github.com/apache/hadoop/pull/282 is stopping this happening in HADOOP-14971.

      Reviews should go into the PR/other task

      1. HADOOP-15033-testfix-1.diff
        2 kB
        Aaron Fabbri
      2. HADOOP-13786-053.patch
        1.02 MB
        Steve Loughran
      3. HADOOP-13786-052.patch
        1.02 MB
        Steve Loughran
      4. HADOOP-13786-051.patch
        1.02 MB
        Steve Loughran
      5. HADOOP-13786-050.patch
        1.01 MB
        Steve Loughran
      6. HADOOP-13786-049.patch
        1.01 MB
        Steve Loughran
      7. HADOOP-13786-048.patch
        1.01 MB
        Steve Loughran
      8. HADOOP-13786-047.patch
        1003 kB
        Steve Loughran
      9. HADOOP-13786-046.patch
        996 kB
        Steve Loughran
      10. HADOOP-13786-045.patch
        993 kB
        Steve Loughran
      11. HADOOP-13786-044.patch
        988 kB
        Steve Loughran
      12. HADOOP-13786-043.patch
        986 kB
        Steve Loughran
      13. HADOOP-13786-042.patch
        983 kB
        Steve Loughran
      14. HADOOP-13786-041.patch
        982 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks, committed under the main JIRA, closing this as a duplicate.

          Show
          stevel@apache.org Steve Loughran added a comment - thanks, committed under the main JIRA, closing this as a duplicate.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 16m 59s trunk passed
          +1 compile 13m 12s trunk passed
          +1 checkstyle 2m 10s trunk passed
          +1 mvnsite 3m 28s trunk passed
          +1 shadedclient 16m 10s branch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 41s trunk passed
          +1 javadoc 2m 47s trunk passed
                Patch Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 34s the patch passed
          +1 compile 11m 35s the patch passed
          +1 javac 11m 35s the patch passed
          -0 checkstyle 2m 4s root: The patch generated 22 new + 122 unchanged - 27 fixed = 144 total (was 149)
          +1 mvnsite 3m 21s the patch passed
          -1 whitespace 0m 0s The patch has 79 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 5s The patch has no ill-formed XML file.
          +1 shadedclient 10m 5s patch has no errors when building and testing our client artifacts.
          +1 findbugs 5m 1s the patch passed
          +1 javadoc 2m 40s the patch passed
                Other Tests
          +1 unit 8m 35s hadoop-common in the patch passed.
          +1 unit 0m 59s hadoop-yarn-registry in the patch passed.
          +1 unit 3m 12s hadoop-mapreduce-client-core in the patch passed.
          -1 unit 23m 0s hadoop-mapreduce-client-app in the patch failed.
          +1 unit 4m 39s hadoop-aws in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          135m 54s



          Reason Tests
          Failed junit tests hadoop.mapreduce.v2.app.TestRecovery



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12898657/HADOOP-13786-053.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux fd828f23a0ab 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 03c311e
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/artifact/out/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/testReport/
          Max. process+thread count 1767 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 16m 59s trunk passed +1 compile 13m 12s trunk passed +1 checkstyle 2m 10s trunk passed +1 mvnsite 3m 28s trunk passed +1 shadedclient 16m 10s branch has no errors when building and testing our client artifacts. +1 findbugs 4m 41s trunk passed +1 javadoc 2m 47s trunk passed       Patch Compile Tests 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 34s the patch passed +1 compile 11m 35s the patch passed +1 javac 11m 35s the patch passed -0 checkstyle 2m 4s root: The patch generated 22 new + 122 unchanged - 27 fixed = 144 total (was 149) +1 mvnsite 3m 21s the patch passed -1 whitespace 0m 0s The patch has 79 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 5s The patch has no ill-formed XML file. +1 shadedclient 10m 5s patch has no errors when building and testing our client artifacts. +1 findbugs 5m 1s the patch passed +1 javadoc 2m 40s the patch passed       Other Tests +1 unit 8m 35s hadoop-common in the patch passed. +1 unit 0m 59s hadoop-yarn-registry in the patch passed. +1 unit 3m 12s hadoop-mapreduce-client-core in the patch passed. -1 unit 23m 0s hadoop-mapreduce-client-app in the patch failed. +1 unit 4m 39s hadoop-aws in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 135m 54s Reason Tests Failed junit tests hadoop.mapreduce.v2.app.TestRecovery Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12898657/HADOOP-13786-053.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux fd828f23a0ab 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 03c311e maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/artifact/out/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/testReport/ Max. process+thread count 1767 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13731/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          My testing in us-west-2 looks good. +1 latest patch.

          Just need to restart jenkins here.

          Show
          fabbri Aaron Fabbri added a comment - My testing in us-west-2 looks good. +1 latest patch. Just need to restart jenkins here.
          Hide
          fabbri Aaron Fabbri added a comment -

          Testing latest patch. Latest commits look good (trivial stuff).

          Show
          fabbri Aaron Fabbri added a comment - Testing latest patch. Latest commits look good (trivial stuff).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 10m 5s Docker failed to build yetus/hadoop:5b98639.



          Subsystem Report/Notes
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12898657/HADOOP-13786-053.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13725/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 10m 5s Docker failed to build yetus/hadoop:5b98639. Subsystem Report/Notes JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12898657/HADOOP-13786-053.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13725/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 053, fix TestTasks checkstyle and tabs in committers.md

          Test, s3 ireland

          I'm pretty much done here

          Show
          stevel@apache.org Steve Loughran added a comment - patch 053, fix TestTasks checkstyle and tabs in committers.md Test, s3 ireland I'm pretty much done here
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Tabs are in console output in committers.md L693; will fix

          Show
          stevel@apache.org Steve Loughran added a comment - Tabs are in console output in committers.md L693; will fix
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 10m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 31s Maven dependency ordering for branch
          +1 mvninstall 16m 7s trunk passed
          +1 compile 12m 16s trunk passed
          +1 checkstyle 2m 6s trunk passed
          +1 mvnsite 3m 24s trunk passed
          +1 shadedclient 15m 47s branch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 23s trunk passed
          +1 javadoc 2m 36s trunk passed
                Patch Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 30s the patch passed
          +1 compile 11m 27s the patch passed
          +1 javac 11m 27s the patch passed
          -0 checkstyle 2m 5s root: The patch generated 23 new + 122 unchanged - 27 fixed = 145 total (was 149)
          +1 mvnsite 3m 23s the patch passed
          -1 whitespace 0m 0s The patch has 79 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 2s The patch 3 line(s) with tabs.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          +1 shadedclient 9m 59s patch has no errors when building and testing our client artifacts.
          +1 findbugs 5m 4s the patch passed
          +1 javadoc 2m 38s the patch passed
                Other Tests
          +1 unit 8m 8s hadoop-common in the patch passed.
          +1 unit 0m 59s hadoop-yarn-registry in the patch passed.
          +1 unit 3m 11s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 8m 48s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 4m 36s hadoop-aws in the patch passed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          131m 15s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12898507/HADOOP-13786-052.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 544f0b39b343 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0d781dd
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/artifact/out/whitespace-tabs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/testReport/
          Max. process+thread count 1523 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 1m 31s Maven dependency ordering for branch +1 mvninstall 16m 7s trunk passed +1 compile 12m 16s trunk passed +1 checkstyle 2m 6s trunk passed +1 mvnsite 3m 24s trunk passed +1 shadedclient 15m 47s branch has no errors when building and testing our client artifacts. +1 findbugs 4m 23s trunk passed +1 javadoc 2m 36s trunk passed       Patch Compile Tests 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 30s the patch passed +1 compile 11m 27s the patch passed +1 javac 11m 27s the patch passed -0 checkstyle 2m 5s root: The patch generated 23 new + 122 unchanged - 27 fixed = 145 total (was 149) +1 mvnsite 3m 23s the patch passed -1 whitespace 0m 0s The patch has 79 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 2s The patch 3 line(s) with tabs. +1 xml 0m 6s The patch has no ill-formed XML file. +1 shadedclient 9m 59s patch has no errors when building and testing our client artifacts. +1 findbugs 5m 4s the patch passed +1 javadoc 2m 38s the patch passed       Other Tests +1 unit 8m 8s hadoop-common in the patch passed. +1 unit 0m 59s hadoop-yarn-registry in the patch passed. +1 unit 3m 11s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 48s hadoop-mapreduce-client-app in the patch passed. +1 unit 4m 36s hadoop-aws in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 131m 15s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12898507/HADOOP-13786-052.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 544f0b39b343 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 0d781dd maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/artifact/out/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/testReport/ Max. process+thread count 1523 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13719/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Note also: minimal downstream unit tests to verify that you can load the standard classes without needing mapreduce JARs on the CP (indeed, the tests makes sure that they aren't, to stop false positive test run). These will be part of my s3a regression testing from now on

          https://github.com/hortonworks-spark/cloud-integration/tree/master/check-cp

          Show
          stevel@apache.org Steve Loughran added a comment - Note also: minimal downstream unit tests to verify that you can load the standard classes without needing mapreduce JARs on the CP (indeed, the tests makes sure that they aren't, to stop false positive test run). These will be part of my s3a regression testing from now on https://github.com/hortonworks-spark/cloud-integration/tree/master/check-cp
          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-13786 patch 052: TestTasks

          • fix log message pointed out by Aaron
          • debug failing test on jenkins; managed to replicate locally by bumping up the #of threads & work, added more logging, identified problem and fixed. It was a bug in TestTasks: ordering of execution meant the item which failed in its execution wasn't always being identified. Adding a new "failed" field addresses that once the assertions are tweaked to match

          Tested: S3 ireland, dynamodb, scale

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-13786 patch 052: TestTasks fix log message pointed out by Aaron debug failing test on jenkins; managed to replicate locally by bumping up the #of threads & work, added more logging, identified problem and fixed. It was a bug in TestTasks: ordering of execution meant the item which failed in its execution wasn't always being identified. Adding a new "failed" field addresses that once the assertions are tweaked to match Tested: S3 ireland, dynamodb, scale
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Test failure looks legit & presumably concurrency; managed to replicate by significantly increasing the #of threads and tasks

          2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-0] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=1, limit=8, item=Item{id=1, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=1, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=1, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-1] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=2, limit=8, item=Item{id=2, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=2, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=2, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-2] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=3, limit=8, item=Item{id=3, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=3, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=3, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-3] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=4, limit=8, item=Item{id=4, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=4, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=4, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-4] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=5, limit=8, item=Item{id=5, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=5, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=5, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-5] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=6, limit=8, item=Item{id=6, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=6, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=6, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-6] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=7, limit=8, item=Item{id=7, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=7, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=7, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-0] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=8, limit=8, item=Item{id=9, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=9, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=9, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-5] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=15, limit=8, item=Item{id=15, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=15, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=15, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-4] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=14, limit=8, item=Item{id=14, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=14, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=14, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-3] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=13, limit=8, item=Item{id=13, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=13, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=13, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-6] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=12, limit=8, item=Item{id=12, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=12, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=12, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-2] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=11, limit=8, item=Item{id=12, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=11, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=11, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-7] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=10, limit=8, item=Item{id=11, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=8, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=8, committed=true, aborted=false, reverted=false, text=With 8 threads}
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-1] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=9, limit=8, item=Item{id=10, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=10, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=10, committed=true, aborted=false, reverted=false, text=With 8 threads}
          
          /** this is the failing task */
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-5] INFO  commit.Tasks (Tasks.java:run(287)) - Aborting task
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-0] INFO  commit.Tasks (Tasks.java:run(263)) - Task failed
          java.io.IOException: BaseCounter{name='failing committer', count=15, limit=8, item=Item{id=15, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Limit 8 reached for Item{id=9, committed=true, aborted=false, reverted=false, text=With 8 threads}
          	at org.apache.hadoop.fs.s3a.commit.TestTasks$BaseCounter.process(TestTasks.java:460)
          	at org.apache.hadoop.fs.s3a.commit.TestTasks$CounterTask.run(TestTasks.java:509)
          	at org.apache.hadoop.fs.s3a.commit.TestTasks$CounterTask.run(TestTasks.java:499)
          	at org.apache.hadoop.fs.s3a.commit.Tasks$Builder$1.run(Tasks.java:254)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-5] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='aborter', count=1, limit=0, item=Item{id=16, committed=false, aborted=true, reverted=false, text=With 8 threads}}: Item{id=16, committed=false, aborted=false, reverted=false, text=With 8 threads} -> Item{id=16, committed=false, aborted=true, reverted=false, text=With 8 threads}
          
          /* this is task ID #9 still being executed */
          
          2017-11-20 15:30:41,311 [testFailFastCallRevertSuppressed[3]-pool-0] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failures', count=1, limit=0, item=Item{id=9, committed=true, aborted=false, reverted=false, text=With 8 threads}}: Item{id=9, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=9, committed=true, aborted=false, reverted=false, text=With 8 threads}
          
          /* reverter starts, but only has 14 successful tasks: it hasn't picked up #9 yet */
          
          2017-11-20 15:30:41,321 [JUnit-testFailFastCallRevertSuppressed[3]] INFO  commit.Tasks (Tasks.java:runParallel(309)) - Reverting all 14 succeeded tasks
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-4] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=1, limit=0, item=Item{id=1, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=1, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=1, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-2] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=4, limit=0, item=Item{id=4, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=4, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=4, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-6] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=3, limit=0, item=Item{id=3, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=3, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=3, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-3] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=2, limit=0, item=Item{id=2, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=2, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=2, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-6] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=11, limit=0, item=Item{id=12, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=12, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=12, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-2] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=10, limit=0, item=Item{id=13, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=13, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=13, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-0] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=9, limit=0, item=Item{id=14, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=14, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=14, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-5] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=8, limit=0, item=Item{id=7, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=7, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=7, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-7] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=6, limit=0, item=Item{id=6, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=5, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=5, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-4] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=7, limit=0, item=Item{id=7, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=15, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=15, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-1] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=6, limit=0, item=Item{id=6, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=6, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=6, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-2] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=14, limit=0, item=Item{id=10, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=10, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=10, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-6] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=13, limit=0, item=Item{id=8, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=8, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=8, committed=true, aborted=false, reverted=true, text=With 8 threads}
          2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-3] INFO  commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=12, limit=0, item=Item{id=11, committed=true, aborted=false, reverted=true, text=With 8 threads}}: Item{id=11, committed=true, aborted=false, reverted=false, text=With 8 threads} -> Item{id=11, committed=true, aborted=false, reverted=true, text=With 8 threads}
          
          /* so the assertion: "all committed tasks were reverted " doesn't hold. */
          java.lang.AssertionError: Item{id=9, committed=true, aborted=false, reverted=false, text=With 8 threads} was not reverted in [Item{id=1, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=2, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=3, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=4, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=5, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=6, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=7, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=8, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=9, committed=true, aborted=false, reverted=false, text=With 8 threads}
          Item{id=10, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=11, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=12, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=13, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=14, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=15, committed=true, aborted=false, reverted=true, text=With 8 threads}
          Item{id=16, committed=false, aborted=true, reverted=false, text=With 8 threads}]
          
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.apache.hadoop.fs.s3a.commit.TestTasks$Item.assertReverted(TestTasks.java:402)
          	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
          	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
          	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
          	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
          	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
          	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
          	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
          	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
          	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
          	at org.apache.hadoop.fs.s3a.commit.TestTasks.testFailFastCallRevertSuppressed(TestTasks.java:231)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Show
          stevel@apache.org Steve Loughran added a comment - Test failure looks legit & presumably concurrency; managed to replicate by significantly increasing the #of threads and tasks 2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-0] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=1, limit=8, item=Item{id=1, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=1, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=1, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-1] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=2, limit=8, item=Item{id=2, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=2, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=2, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-2] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=3, limit=8, item=Item{id=3, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=3, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=3, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-3] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=4, limit=8, item=Item{id=4, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=4, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=4, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,309 [testFailFastCallRevertSuppressed[3]-pool-4] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=5, limit=8, item=Item{id=5, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=5, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=5, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-5] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=6, limit=8, item=Item{id=6, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=6, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=6, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-6] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=7, limit=8, item=Item{id=7, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=7, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=7, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-0] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=8, limit=8, item=Item{id=9, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=9, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=9, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-5] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=15, limit=8, item=Item{id=15, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=15, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=15, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-4] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=14, limit=8, item=Item{id=14, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=14, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=14, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-3] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=13, limit=8, item=Item{id=13, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=13, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=13, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-6] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=12, limit=8, item=Item{id=12, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=12, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=12, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-2] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=11, limit=8, item=Item{id=12, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=11, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=11, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-7] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=10, limit=8, item=Item{id=11, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=8, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=8, committed= true , aborted= false , reverted= false , text=With 8 threads} 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-1] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failing committer', count=9, limit=8, item=Item{id=10, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=10, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=10, committed= true , aborted= false , reverted= false , text=With 8 threads} /** this is the failing task */ 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-5] INFO commit.Tasks (Tasks.java:run(287)) - Aborting task 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-0] INFO commit.Tasks (Tasks.java:run(263)) - Task failed java.io.IOException: BaseCounter{name='failing committer', count=15, limit=8, item=Item{id=15, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Limit 8 reached for Item{id=9, committed= true , aborted= false , reverted= false , text=With 8 threads} at org.apache.hadoop.fs.s3a.commit.TestTasks$BaseCounter.process(TestTasks.java:460) at org.apache.hadoop.fs.s3a.commit.TestTasks$CounterTask.run(TestTasks.java:509) at org.apache.hadoop.fs.s3a.commit.TestTasks$CounterTask.run(TestTasks.java:499) at org.apache.hadoop.fs.s3a.commit.Tasks$Builder$1.run(Tasks.java:254) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) 2017-11-20 15:30:41,310 [testFailFastCallRevertSuppressed[3]-pool-5] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='aborter', count=1, limit=0, item=Item{id=16, committed= false , aborted= true , reverted= false , text=With 8 threads}}: Item{id=16, committed= false , aborted= false , reverted= false , text=With 8 threads} -> Item{id=16, committed= false , aborted= true , reverted= false , text=With 8 threads} /* this is task ID #9 still being executed */ 2017-11-20 15:30:41,311 [testFailFastCallRevertSuppressed[3]-pool-0] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='failures', count=1, limit=0, item=Item{id=9, committed= true , aborted= false , reverted= false , text=With 8 threads}}: Item{id=9, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=9, committed= true , aborted= false , reverted= false , text=With 8 threads} /* reverter starts, but only has 14 successful tasks: it hasn't picked up #9 yet */ 2017-11-20 15:30:41,321 [JUnit-testFailFastCallRevertSuppressed[3]] INFO commit.Tasks (Tasks.java:runParallel(309)) - Reverting all 14 succeeded tasks 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-4] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=1, limit=0, item=Item{id=1, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=1, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=1, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-2] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=4, limit=0, item=Item{id=4, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=4, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=4, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-6] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=3, limit=0, item=Item{id=3, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=3, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=3, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-3] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=2, limit=0, item=Item{id=2, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=2, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=2, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-6] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=11, limit=0, item=Item{id=12, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=12, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=12, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-2] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=10, limit=0, item=Item{id=13, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=13, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=13, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-0] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=9, limit=0, item=Item{id=14, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=14, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=14, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-5] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=8, limit=0, item=Item{id=7, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=7, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=7, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-7] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=6, limit=0, item=Item{id=6, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=5, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=5, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-4] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=7, limit=0, item=Item{id=7, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=15, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=15, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-1] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=6, limit=0, item=Item{id=6, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=6, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=6, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-2] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=14, limit=0, item=Item{id=10, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=10, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=10, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-6] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=13, limit=0, item=Item{id=8, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=8, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=8, committed= true , aborted= false , reverted= true , text=With 8 threads} 2017-11-20 15:30:41,322 [testFailFastCallRevertSuppressed[3]-pool-3] INFO commit.TestTasks (TestTasks.java:process(458)) - BaseCounter{name='reverter', count=12, limit=0, item=Item{id=11, committed= true , aborted= false , reverted= true , text=With 8 threads}}: Item{id=11, committed= true , aborted= false , reverted= false , text=With 8 threads} -> Item{id=11, committed= true , aborted= false , reverted= true , text=With 8 threads} /* so the assertion: "all committed tasks were reverted " doesn't hold. */ java.lang.AssertionError: Item{id=9, committed= true , aborted= false , reverted= false , text=With 8 threads} was not reverted in [Item{id=1, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=2, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=3, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=4, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=5, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=6, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=7, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=8, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=9, committed= true , aborted= false , reverted= false , text=With 8 threads} Item{id=10, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=11, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=12, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=13, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=14, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=15, committed= true , aborted= false , reverted= true , text=With 8 threads} Item{id=16, committed= false , aborted= true , reverted= false , text=With 8 threads}] at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.fs.s3a.commit.TestTasks$Item.assertReverted(TestTasks.java:402) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.apache.hadoop.fs.s3a.commit.TestTasks.testFailFastCallRevertSuppressed(TestTasks.java:231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Only one relevant test failure, TestTasks

          Tests run: 36, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.935 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.TestTasks
          testFailFastCallRevertSuppressed[2](org.apache.hadoop.fs.s3a.commit.TestTasks)  Time elapsed: 0.027 sec  <<< FAILURE!
          java.lang.AssertionError: Operation{id=3, committed=true, aborted=false, reverted=false} was not reverted
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.apache.hadoop.fs.s3a.commit.TestTasks$Item.assertReverted(TestTasks.java:390)
          	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
          	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
          	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380)
          	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
          	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
          	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
          	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
          	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
          	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
          	at org.apache.hadoop.fs.s3a.commit.TestTasks.testFailFastCallRevertSuppressed(TestTasks.java:223)
          

          FWIW, I like that Tasks stuff enough to think "it should be pooled into S3A itself & used for the complex operations there, or even -> hadoop-common". not worrying about it for this patch thoiuch.

          Show
          stevel@apache.org Steve Loughran added a comment - Only one relevant test failure, TestTasks Tests run: 36, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.935 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.TestTasks testFailFastCallRevertSuppressed[2](org.apache.hadoop.fs.s3a.commit.TestTasks) Time elapsed: 0.027 sec <<< FAILURE! java.lang.AssertionError: Operation{id=3, committed= true , aborted= false , reverted= false } was not reverted at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.fs.s3a.commit.TestTasks$Item.assertReverted(TestTasks.java:390) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.apache.hadoop.fs.s3a.commit.TestTasks.testFailFastCallRevertSuppressed(TestTasks.java:223) FWIW, I like that Tasks stuff enough to think "it should be pooled into S3A itself & used for the complex operations there, or even -> hadoop-common". not worrying about it for this patch thoiuch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 9m 45s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 20m 20s trunk passed
          +1 compile 16m 44s trunk passed
          +1 checkstyle 2m 11s trunk passed
          +1 mvnsite 4m 13s trunk passed
          +1 shadedclient 17m 2s branch has no errors when building and testing our client artifacts.
          +1 findbugs 6m 58s trunk passed
          +1 javadoc 2m 58s trunk passed
                Patch Compile Tests
          0 mvndep 0m 22s Maven dependency ordering for patch
          +1 mvninstall 4m 10s the patch passed
          +1 compile 18m 7s the patch passed
          +1 javac 18m 7s the patch passed
          -0 checkstyle 2m 22s root: The patch generated 22 new + 122 unchanged - 27 fixed = 144 total (was 149)
          +1 mvnsite 4m 15s the patch passed
          -1 whitespace 0m 0s The patch has 80 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 2s The patch 3 line(s) with tabs.
          +1 xml 0m 10s The patch has no ill-formed XML file.
          +1 shadedclient 10m 39s patch has no errors when building and testing our client artifacts.
          +1 findbugs 7m 19s the patch passed
          +1 javadoc 2m 55s the patch passed
                Other Tests
          +1 unit 10m 40s hadoop-common in the patch passed.
          +1 unit 0m 59s hadoop-yarn-registry in the patch passed.
          +1 unit 3m 45s hadoop-mapreduce-client-core in the patch passed.
          -1 unit 23m 53s hadoop-mapreduce-client-app in the patch failed.
          -1 unit 4m 33s hadoop-aws in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          172m 13s



          Reason Tests
          Unreaped Processes hadoop-mapreduce-client-app:1
          Failed junit tests hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator
            hadoop.fs.s3a.commit.TestTasks



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12898266/HADOOP-13786-051.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 868d0485d6df 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0940e4f
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/whitespace-tabs.txt
          Unreaped Processes Log https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-reaper.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/testReport/
          Max. process+thread count 1501 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 9m 45s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 20m 20s trunk passed +1 compile 16m 44s trunk passed +1 checkstyle 2m 11s trunk passed +1 mvnsite 4m 13s trunk passed +1 shadedclient 17m 2s branch has no errors when building and testing our client artifacts. +1 findbugs 6m 58s trunk passed +1 javadoc 2m 58s trunk passed       Patch Compile Tests 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 4m 10s the patch passed +1 compile 18m 7s the patch passed +1 javac 18m 7s the patch passed -0 checkstyle 2m 22s root: The patch generated 22 new + 122 unchanged - 27 fixed = 144 total (was 149) +1 mvnsite 4m 15s the patch passed -1 whitespace 0m 0s The patch has 80 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 2s The patch 3 line(s) with tabs. +1 xml 0m 10s The patch has no ill-formed XML file. +1 shadedclient 10m 39s patch has no errors when building and testing our client artifacts. +1 findbugs 7m 19s the patch passed +1 javadoc 2m 55s the patch passed       Other Tests +1 unit 10m 40s hadoop-common in the patch passed. +1 unit 0m 59s hadoop-yarn-registry in the patch passed. +1 unit 3m 45s hadoop-mapreduce-client-core in the patch passed. -1 unit 23m 53s hadoop-mapreduce-client-app in the patch failed. -1 unit 4m 33s hadoop-aws in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 172m 13s Reason Tests Unreaped Processes hadoop-mapreduce-client-app:1 Failed junit tests hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator   hadoop.fs.s3a.commit.TestTasks Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12898266/HADOOP-13786-051.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 868d0485d6df 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 0940e4f maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/whitespace-tabs.txt Unreaped Processes Log https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-reaper.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/testReport/ Max. process+thread count 1501 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13712/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 051

          Test failure from reuse of non-magic FS.

          • special check in file creation for __magic path and not magic fs, logs @ info. Warn maybe?
          • committer troubleshooting mentions what this warning means
          • testing.md lists what we want from good tests (parallelise, etc), and mentions FS caching
          • magic tests disable FS Caching.

          we may want to simplify things here by always setting the magic build in our test filesystems. That way: no need to play these games and no recurrent problems. Thoughts?

          Isolate S3A filesystem from transitive dependencies on hadoop-mapreduce

          Isolate all utility classes which use MR datatypes from the others into their own package, and the magic path ones, splitting CommitUtils in three. This is needed to ensure s3a loads when mapreduce JARs aren't on the classpath.

          The previous patches probably meant S3AFS woudn't load without hadoop-mapreduce on the CP just because some other static methods in CommitUtils needed it. Partitioning things up and a full review of the s3 codepath appears to fix this. However, I plan to add a new downstream test module somewhere just to verify classpath setup by opening the landsat repository and listing it,

          Show
          stevel@apache.org Steve Loughran added a comment - patch 051 Test failure from reuse of non-magic FS. special check in file creation for __magic path and not magic fs, logs @ info. Warn maybe? committer troubleshooting mentions what this warning means testing.md lists what we want from good tests (parallelise, etc), and mentions FS caching magic tests disable FS Caching. we may want to simplify things here by always setting the magic build in our test filesystems. That way: no need to play these games and no recurrent problems. Thoughts? Isolate S3A filesystem from transitive dependencies on hadoop-mapreduce Isolate all utility classes which use MR datatypes from the others into their own package, and the magic path ones, splitting CommitUtils in three. This is needed to ensure s3a loads when mapreduce JARs aren't on the classpath. The previous patches probably meant S3AFS woudn't load without hadoop-mapreduce on the CP just because some other static methods in CommitUtils needed it. Partitioning things up and a full review of the s3 codepath appears to fix this. However, I plan to add a new downstream test module somewhere just to verify classpath setup by opening the landsat repository and listing it,
          Hide
          fabbri Aaron Fabbri added a comment -

          Good point about this being a potential failure mode. +1 adding scary log messages on __magic path w/o feature enabled.

          This test is marked to run in serial test section. I assume the FS reuse pattern is still somehow influenced by the different sequence of runs when I did parallel tests. So the parallel thing is, on one hand, unrelated, but still has side effects that allowed me to reproduce it.

          Anyways, I assume you have another patch coming? Will catch up on the github pull request additions meanwhile.

          Show
          fabbri Aaron Fabbri added a comment - Good point about this being a potential failure mode. +1 adding scary log messages on __magic path w/o feature enabled. This test is marked to run in serial test section. I assume the FS reuse pattern is still somehow influenced by the different sequence of runs when I did parallel tests. So the parallel thing is, on one hand, unrelated, but still has side effects that allowed me to reproduce it. Anyways, I assume you have another patch coming? Will catch up on the github pull request additions meanwhile.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          FWIW, this is a sign of what will be a problem: s3a magic committer being used when the FS doesn't have it turned on.

          I'd made it an option as I didn't want to add special behaviour in case anyone was using the paths, and because you shouldn't be using it without s3guard enabled.

          If it happens now, it'll happen in future.

          Immediate changes

          • special warning if you try to create a file with a magic path and the FS is not.
          • section in troubleshooting committers

          FWIW, I am running my test FS with magic enabled, as I need it for other tests...that'll be why I never saw the problem but you did. I'll switch to a different bucket for tests here

          Show
          stevel@apache.org Steve Loughran added a comment - FWIW, this is a sign of what will be a problem: s3a magic committer being used when the FS doesn't have it turned on. I'd made it an option as I didn't want to add special behaviour in case anyone was using the paths, and because you shouldn't be using it without s3guard enabled. If it happens now, it'll happen in future. Immediate changes special warning if you try to create a file with a magic path and the FS is not. section in troubleshooting committers FWIW, I am running my test FS with magic enabled, as I need it for other tests...that'll be why I never saw the problem but you did. I'll switch to a different bucket for tests here
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Oh, it'll be VM recycling of filesystems. Same reason some of the encryption tests are in the serial bit. But I'd thought this test was also set up for serial use

          Anyway, good to see this was a test related problem...and last week it forced me to go through all the cleanup code anyway, which is now cleaned up.

          I'll apply the patch & test, plus see about adding a sentence about FS recycling in parallel tests

          Show
          stevel@apache.org Steve Loughran added a comment - Oh, it'll be VM recycling of filesystems. Same reason some of the encryption tests are in the serial bit. But I'd thought this test was also set up for serial use Anyway, good to see this was a test related problem...and last week it forced me to go through all the cleanup code anyway, which is now cleaned up. I'll apply the patch & test, plus see about adding a sentence about FS recycling in parallel tests
          Hide
          fabbri Aaron Fabbri added a comment -

          (Attaching a small patch.)

          Busy day today but I think I have a fix for the parallel test failure. What I did:

          • Instrumented test class configuration constructors/getters.
          • Dumped a stack trace on S3AFileSystem#initialize().
          • Added the isMagicCommitEnabled bit to S3AFileSystem#toString() (included in patch).
          • Reproduced the failure.

          What I saw:

          • Beginning part of huge scale test, where the large file is created, was using some existing S3AFileSystem instance with isMagicCommitEnabled false, despite us doing the work to set up a Configuration with the option set to true.
          • Second part of huge test (post create assertions) that was failing on asserting that the .pending file was there was getting a new S3AFileSystem with the magic commit flag enabled, but it didn't matter since the creation did not have it enabled.

          Seems like the initial FS is coming from the FileSystem cache so I disabled that (patch included) and re-ran and my reproduction vanished. Logging shows that both test phases (create, post assert) are getting a fresh FileSystem with the magic commit flag enabled now.

          Show
          fabbri Aaron Fabbri added a comment - (Attaching a small patch.) Busy day today but I think I have a fix for the parallel test failure. What I did: Instrumented test class configuration constructors/getters. Dumped a stack trace on S3AFileSystem#initialize() . Added the isMagicCommitEnabled bit to S3AFileSystem#toString() (included in patch). Reproduced the failure. What I saw: Beginning part of huge scale test, where the large file is created, was using some existing S3AFileSystem instance with isMagicCommitEnabled false, despite us doing the work to set up a Configuration with the option set to true. Second part of huge test (post create assertions) that was failing on asserting that the .pending file was there was getting a new S3AFileSystem with the magic commit flag enabled, but it didn't matter since the creation did not have it enabled. Seems like the initial FS is coming from the FileSystem cache so I disabled that (patch included) and re-ran and my reproduction vanished. Logging shows that both test phases (create, post assert) are getting a fresh FileSystem with the magic commit flag enabled now.
          Hide
          fabbri Aaron Fabbri added a comment -

          The cause of the ITestS3AHugeMagicCommits failure I'm seeing now is that the Configuration object does not have magic commit enabled. This causes us to create a normal (non-magic) put tracker. Haven't tracked down the cause of the Configuration issue yet. Again the subclassing in these tests makes it a bit obfuscated. It looks like we attempt to enable magic commits in both AbstractS3AScaleTestBase#createConfiguration() and in ITestS3AHugeMagicCommits#createScaleConfiguration()

          Show
          fabbri Aaron Fabbri added a comment - The cause of the ITestS3AHugeMagicCommits failure I'm seeing now is that the Configuration object does not have magic commit enabled. This causes us to create a normal (non-magic) put tracker. Haven't tracked down the cause of the Configuration issue yet. Again the subclassing in these tests makes it a bit obfuscated. It looks like we attempt to enable magic commits in both AbstractS3AScaleTestBase#createConfiguration() and in ITestS3AHugeMagicCommits#createScaleConfiguration()
          Hide
          fabbri Aaron Fabbri added a comment -

          FYI looks like I may have reproduced the parallel test failure with better logging.. it is late here I will debug it in the morning.

          Show
          fabbri Aaron Fabbri added a comment - FYI looks like I may have reproduced the parallel test failure with better logging.. it is late here I will debug it in the morning.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 15m 55s trunk passed
          +1 compile 11m 49s trunk passed
          +1 checkstyle 2m 4s trunk passed
          +1 mvnsite 3m 18s trunk passed
          +1 shadedclient 15m 42s branch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 20s trunk passed
          +1 javadoc 2m 34s trunk passed
                Patch Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 29s the patch passed
          +1 compile 11m 2s the patch passed
          +1 javac 11m 2s the patch passed
          -0 checkstyle 2m 5s root: The patch generated 23 new + 117 unchanged - 22 fixed = 140 total (was 139)
          +1 mvnsite 3m 18s the patch passed
          -1 whitespace 0m 0s The patch has 75 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 2s The patch 3 line(s) with tabs.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          +1 shadedclient 10m 0s patch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 57s the patch passed
          +1 javadoc 2m 33s the patch passed
                Other Tests
          +1 unit 8m 13s hadoop-common in the patch passed.
          +1 unit 0m 53s hadoop-yarn-registry in the patch passed.
          +1 unit 2m 55s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 8m 51s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 4m 42s hadoop-aws in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          116m 26s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897791/HADOOP-13786-050.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux a2f00af214f1 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / fac72ee
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/artifact/out/whitespace-tabs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/testReport/
          Max. process+thread count 1702 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 15m 55s trunk passed +1 compile 11m 49s trunk passed +1 checkstyle 2m 4s trunk passed +1 mvnsite 3m 18s trunk passed +1 shadedclient 15m 42s branch has no errors when building and testing our client artifacts. +1 findbugs 4m 20s trunk passed +1 javadoc 2m 34s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 29s the patch passed +1 compile 11m 2s the patch passed +1 javac 11m 2s the patch passed -0 checkstyle 2m 5s root: The patch generated 23 new + 117 unchanged - 22 fixed = 140 total (was 139) +1 mvnsite 3m 18s the patch passed -1 whitespace 0m 0s The patch has 75 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 2s The patch 3 line(s) with tabs. +1 xml 0m 6s The patch has no ill-formed XML file. +1 shadedclient 10m 0s patch has no errors when building and testing our client artifacts. +1 findbugs 4m 57s the patch passed +1 javadoc 2m 33s the patch passed       Other Tests +1 unit 8m 13s hadoop-common in the patch passed. +1 unit 0m 53s hadoop-yarn-registry in the patch passed. +1 unit 2m 55s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 51s hadoop-mapreduce-client-app in the patch passed. +1 unit 4m 42s hadoop-aws in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 116m 26s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897791/HADOOP-13786-050.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux a2f00af214f1 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / fac72ee maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/artifact/out/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/testReport/ Max. process+thread count 1702 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13689/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          test: s3 ireland w ddb in auth & non-auth

          Show
          stevel@apache.org Steve Loughran added a comment - test: s3 ireland w ddb in auth & non-auth
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 050 typos and checkstyle: encapsulate field,, suppress deprecation warning in subclasses, plus a couple of spelling errors.

          verified the hadoop common test was passing again

          Show
          stevel@apache.org Steve Loughran added a comment - patch 050 typos and checkstyle: encapsulate field,, suppress deprecation warning in subclasses, plus a couple of spelling errors. verified the hadoop common test was passing again
          Hide
          stevel@apache.org Steve Loughran added a comment -

          typo in the core-defaults, "s.s3a.committer.staging.abort.pending.uploads"

          Show
          stevel@apache.org Steve Loughran added a comment - typo in the core-defaults, "s.s3a.committer.staging.abort.pending.uploads"
          Hide
          stevel@apache.org Steve Loughran added a comment -

          TestCommonConfigurationFields that's the one which uses introspection to identify properties in core-default.xml which aren't matched in CommonConfigurationKeys & others in hadoop-common JAR. fs.s3a.* should be skipped automatically.

          I did make changes to its superclass TestConfigurationFieldsBase recently to track down a problem, so if there's a regression, it ll be me.

          w.r.t s3guard, there's a couple of failures there; another one is HADOOP-14946, and HADOOP-15022 covers how the table capacity is going up by 1 every test run if your fs.s3a.test bucket is s3guarded, and a move to java 8 could clean up bits of the code. How about I do that sometime as a quick cleanup on the side, along with a look at the unrelated checkstyle warnings we see here. Maybe as part of a review of the s3guard retry logic?

          Show
          stevel@apache.org Steve Loughran added a comment - TestCommonConfigurationFields that's the one which uses introspection to identify properties in core-default.xml which aren't matched in CommonConfigurationKeys & others in hadoop-common JAR. fs.s3a.* should be skipped automatically. I did make changes to its superclass TestConfigurationFieldsBase recently to track down a problem, so if there's a regression, it ll be me. w.r.t s3guard, there's a couple of failures there; another one is HADOOP-14946 , and HADOOP-15022 covers how the table capacity is going up by 1 every test run if your fs.s3a.test bucket is s3guarded, and a move to java 8 could clean up bits of the code. How about I do that sometime as a quick cleanup on the side, along with a look at the unrelated checkstyle warnings we see here. Maybe as part of a review of the s3guard retry logic?
          Hide
          fabbri Aaron Fabbri added a comment -

          I completed 20 runs of scale and integration tests last night on v48 patch. Only failures were related to HADOOP-14927.

          Show
          fabbri Aaron Fabbri added a comment - I completed 20 runs of scale and integration tests last night on v48 patch. Only failures were related to HADOOP-14927 .
          Hide
          fabbri Aaron Fabbri added a comment -

          Javadoc: First one looks like an infra issue. Second one I cannot view. Third one appears to be existing code?
          Javac: deprecation warnings, unavoidable.
          Checkstyle: mostly combination of javadoc links that cannot be broken into multi-line (AFAIK), and existing warnings in adjacent code (that we should fix at some point).
          Unit: TestCommonConfigurationFields. Is there an existing JIRA for this? I don't recall you modifying this part of the code.

          Show
          fabbri Aaron Fabbri added a comment - Javadoc: First one looks like an infra issue. Second one I cannot view. Third one appears to be existing code? Javac: deprecation warnings, unavoidable. Checkstyle: mostly combination of javadoc links that cannot be broken into multi-line (AFAIK), and existing warnings in adjacent code (that we should fix at some point). Unit: TestCommonConfigurationFields. Is there an existing JIRA for this? I don't recall you modifying this part of the code.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 14s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 2m 1s Maven dependency ordering for branch
          +1 mvninstall 31m 40s trunk passed
          +1 compile 26m 24s trunk passed
          +1 checkstyle 3m 30s trunk passed
          +1 mvnsite 7m 28s trunk passed
          +1 shadedclient 26m 33s branch has no errors when building and testing our client artifacts.
          +1 findbugs 10m 5s trunk passed
          -1 javadoc 0m 39s hadoop-mapreduce-client-core in trunk failed.
          -1 javadoc 0m 38s hadoop-mapreduce-client-app in trunk failed.
                Patch Compile Tests
          0 mvndep 0m 28s Maven dependency ordering for patch
          +1 mvninstall 4m 5s the patch passed
          +1 compile 12m 53s the patch passed
          -1 javac 12m 53s root generated 2 new + 1234 unchanged - 0 fixed = 1236 total (was 1234)
          -0 checkstyle 2m 3s root: The patch generated 28 new + 117 unchanged - 22 fixed = 145 total (was 139)
          +1 mvnsite 3m 23s the patch passed
          -1 whitespace 0m 0s The patch has 75 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 1s The patch 3 line(s) with tabs.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          +1 shadedclient 10m 0s patch has no errors when building and testing our client artifacts.
          +1 findbugs 5m 36s the patch passed
          -1 javadoc 0m 32s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 19 new + 0 unchanged - 0 fixed = 19 total (was 0)
                Other Tests
          -1 unit 7m 50s hadoop-common in the patch failed.
          +1 unit 1m 0s hadoop-yarn-registry in the patch passed.
          +1 unit 3m 16s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 9m 10s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 4m 47s hadoop-aws in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          186m 43s



          Reason Tests
          Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897593/HADOOP-13786-049.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 2cd01d9b2f6c 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 18621af
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/whitespace-tabs.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/testReport/
          Max. process+thread count 1471 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 2m 1s Maven dependency ordering for branch +1 mvninstall 31m 40s trunk passed +1 compile 26m 24s trunk passed +1 checkstyle 3m 30s trunk passed +1 mvnsite 7m 28s trunk passed +1 shadedclient 26m 33s branch has no errors when building and testing our client artifacts. +1 findbugs 10m 5s trunk passed -1 javadoc 0m 39s hadoop-mapreduce-client-core in trunk failed. -1 javadoc 0m 38s hadoop-mapreduce-client-app in trunk failed.       Patch Compile Tests 0 mvndep 0m 28s Maven dependency ordering for patch +1 mvninstall 4m 5s the patch passed +1 compile 12m 53s the patch passed -1 javac 12m 53s root generated 2 new + 1234 unchanged - 0 fixed = 1236 total (was 1234) -0 checkstyle 2m 3s root: The patch generated 28 new + 117 unchanged - 22 fixed = 145 total (was 139) +1 mvnsite 3m 23s the patch passed -1 whitespace 0m 0s The patch has 75 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 1s The patch 3 line(s) with tabs. +1 xml 0m 6s The patch has no ill-formed XML file. +1 shadedclient 10m 0s patch has no errors when building and testing our client artifacts. +1 findbugs 5m 36s the patch passed -1 javadoc 0m 32s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 19 new + 0 unchanged - 0 fixed = 19 total (was 0)       Other Tests -1 unit 7m 50s hadoop-common in the patch failed. +1 unit 1m 0s hadoop-yarn-registry in the patch passed. +1 unit 3m 16s hadoop-mapreduce-client-core in the patch passed. +1 unit 9m 10s hadoop-mapreduce-client-app in the patch passed. +1 unit 4m 47s hadoop-aws in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 186m 43s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897593/HADOOP-13786-049.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 2cd01d9b2f6c 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 18621af maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/whitespace-tabs.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/testReport/ Max. process+thread count 1471 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13675/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 049; this is just 048 with those addressable bits of checkstyle addressed.

          we appear to have crossed the 1MB of source milestone. oops

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 049; this is just 048 with those addressable bits of checkstyle addressed. we appear to have crossed the 1MB of source milestone. oops
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks; just gone through the checkstyle and tweaked the new warnings. Still learning about the checkstyle-approved indentation of multiline lambda-exps.

          Show
          stevel@apache.org Steve Loughran added a comment - thanks; just gone through the checkstyle and tweaked the new warnings. Still learning about the checkstyle-approved indentation of multiline lambda-exps.
          Hide
          fabbri Aaron Fabbri added a comment -

          I'm running the latest patch through my "repeat integration and scale tests overnight" script again. I'll post an update tomorrow morning-ish Pacific time.

          Show
          fabbri Aaron Fabbri added a comment - I'm running the latest patch through my "repeat integration and scale tests overnight" script again. I'll post an update tomorrow morning-ish Pacific time.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 16m 5s trunk passed
          +1 compile 11m 51s trunk passed
          +1 checkstyle 2m 2s trunk passed
          +1 mvnsite 3m 17s trunk passed
          +1 shadedclient 15m 51s branch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 19s trunk passed
          +1 javadoc 2m 33s trunk passed
                Patch Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 29s the patch passed
          +1 compile 11m 40s the patch passed
          -1 javac 11m 40s root generated 2 new + 1234 unchanged - 0 fixed = 1236 total (was 1234)
          -0 checkstyle 2m 9s root: The patch generated 55 new + 117 unchanged - 22 fixed = 172 total (was 139)
          +1 mvnsite 3m 32s the patch passed
          -1 whitespace 0m 0s The patch has 75 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 2s The patch 3 line(s) with tabs.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          +1 shadedclient 9m 56s patch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 59s the patch passed
          -1 javadoc 0m 24s hadoop-tools_hadoop-aws generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
                Other Tests
          -1 unit 7m 45s hadoop-common in the patch failed.
          +1 unit 0m 53s hadoop-yarn-registry in the patch passed.
          +1 unit 2m 54s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 8m 36s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 4m 36s hadoop-aws in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          116m 50s



          Reason Tests
          Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897386/HADOOP-13786-048.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 82844b2aa6a4 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 040a38d
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/whitespace-tabs.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/testReport/
          Max. process+thread count 1701 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 16m 5s trunk passed +1 compile 11m 51s trunk passed +1 checkstyle 2m 2s trunk passed +1 mvnsite 3m 17s trunk passed +1 shadedclient 15m 51s branch has no errors when building and testing our client artifacts. +1 findbugs 4m 19s trunk passed +1 javadoc 2m 33s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 29s the patch passed +1 compile 11m 40s the patch passed -1 javac 11m 40s root generated 2 new + 1234 unchanged - 0 fixed = 1236 total (was 1234) -0 checkstyle 2m 9s root: The patch generated 55 new + 117 unchanged - 22 fixed = 172 total (was 139) +1 mvnsite 3m 32s the patch passed -1 whitespace 0m 0s The patch has 75 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 2s The patch 3 line(s) with tabs. +1 xml 0m 6s The patch has no ill-formed XML file. +1 shadedclient 9m 56s patch has no errors when building and testing our client artifacts. +1 findbugs 4m 59s the patch passed -1 javadoc 0m 24s hadoop-tools_hadoop-aws generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)       Other Tests -1 unit 7m 45s hadoop-common in the patch failed. +1 unit 0m 53s hadoop-yarn-registry in the patch passed. +1 unit 2m 54s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 36s hadoop-mapreduce-client-app in the patch passed. +1 unit 4m 36s hadoop-aws in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 116m 50s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897386/HADOOP-13786-048.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 82844b2aa6a4 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 040a38d maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/whitespace-tabs.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/testReport/ Max. process+thread count 1701 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13671/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Yetus didn't run the last patch, so lets try again.

          Changes here minor; apart from staging committer opt out of uploads, & test tuning, this is about documentation and marking up the DDB code for its retry policy. Looks to me like we need to implement exception translation there as well as retrying ops where appropriate.

          • staging committers can opt out of aborting all uploads in job cleanup; consistent with the original netflix committer. This allows >1 partitioned committer to write to the same directory tree as long as you are confident that they really are writing to different partitions (race conditions in parallel job commits, see)
          • Add SocketTimeoutException as retryable on idempotent calls
          • add Retries policy to DynamoDBMetadataStore. This highlights there's little exception translation taking place, and not enough retries. Not added any more retry/translation; that's a S3Guard issue. Just documented what's happening to see what is needed and filed HADOOP-15035.
          • DynamoDB uses same idempotency policy on delete as S3 (i.e. use the same constant in retry(delete))
          • update committer documentation on configuration, more troubleshooting, whole section in index.md on retries and recovery
          • Tests: ITestS3AHugeMagicCommits checks pending upload count before (==1) and after (==0) commit. Didn't find any problems, as expected.
          Show
          stevel@apache.org Steve Loughran added a comment - Yetus didn't run the last patch, so lets try again. Changes here minor; apart from staging committer opt out of uploads, & test tuning, this is about documentation and marking up the DDB code for its retry policy. Looks to me like we need to implement exception translation there as well as retrying ops where appropriate. staging committers can opt out of aborting all uploads in job cleanup; consistent with the original netflix committer. This allows >1 partitioned committer to write to the same directory tree as long as you are confident that they really are writing to different partitions (race conditions in parallel job commits, see) Add SocketTimeoutException as retryable on idempotent calls add Retries policy to DynamoDBMetadataStore. This highlights there's little exception translation taking place, and not enough retries. Not added any more retry/translation; that's a S3Guard issue. Just documented what's happening to see what is needed and filed HADOOP-15035 . DynamoDB uses same idempotency policy on delete as S3 (i.e. use the same constant in retry(delete)) update committer documentation on configuration, more troubleshooting, whole section in index.md on retries and recovery Tests: ITestS3AHugeMagicCommits checks pending upload count before (==1) and after (==0) commit. Didn't find any problems, as expected.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Aarons test failure has forced me to look at all the abort code in the committers. So I have.

          • Found bug in the existing S3AFileSystem.listPendingUploads(path), in that as path never had a trailing "/", it'd list pending uploads in any directory which began with the same prefix, e.g. /work/job1 and work/job12. The final committer cleanup() logic would therefore abort adjacent work. Fixed.
          • turned off that bulk cleanup to see what would happen, helped explore that list and delete pending code, which ultimately proved insufficent (see below). listPendingUploads-> abort is in fact the only way to do it reliably.
          • logging improved @debug level, with a bit more on impoort to deal with the use case "things aren't working in production, here are the logs, VM is gone"

          Two little issues related to cleanup

          Magic App attempt abort would list all in app attempt dest/_magic/$jobid$attemptid/{*.pendingset, *.pending} and clean up
          But this doesn't cleanup on any failed attempt, which will have a different attemptID, so not be found.

          For the magic committer, switched to a much simpler cleanup of

          • list all uploads pending under $dest, abort
          • magic: delete __magic/*
          • staging: abort wrapped job
          • staging: delete local staging dirs for entire job

          That is: no list/read/cancel of pending job & task events. Expensive and not complete.
          This bulk abort of all pending uploads of a dest handles: failed tasks, failed attempts, failure of incomplete jobs from other processes/work which was never aborted. (e.g: a previous query somewhere failed leaving pending work).
          This was already being done after all the "structured" aborts; here I remove that work and only do the bulk list & rm for a simpler life; made sure all of that excutes in parallel for fastest cleanup.

          This significantly simplifies the abort, commit failure and helper classes (no need to suppress exceptions),
          and reduces the diff between the staging & magic committers (so more has been pulled up to the base class)

          That is: this is the first patch for a while which cuts out production code in any measurable way.

          The staging committer currently does the same, but it retains the listing and abort of all outstanding requests first. Why? I've kept it there to handle the use case of >1 partitioned commit running simultaneously, where you don't want to abort all the outstanding stuff. That's not fully rounded off in this patch because it's still doing the bulk cleanup: I want to see what Ryan thinks is best here. I think I'm going to opt for: list then optional bulk abort. That means the default doesn't leave pending objects around. That makes the list operation superfluous, of course, but (a) it's going against HDFS so is less expensive and (b) means that all tests cover the codepath, rather than having a condition which has weak coverage. That is: I don't think it's that harmful: there's the cost of an HDFS list(recursive=true), an open + parse of every read; For the magic committer, in constrast, its more expensive (the LIST, the GET) and it doesn't support that notion of partitioned work: you can't get >1 job writing to the same directory tree.

          Docs:

          • mention examining the _SUCCESS file in troubleshooting section
          • defaults of the various options

          I'm wrapping up some details on cleanup in the committer arch doc, not included in this patch as it is late on a friday evening.

          Misc

          • Invoke adds new method ignoreIOExceptions(Log, text, path, lambda-operation) which runs the operation & logs & info if there is a problem. This replaces a lot of the try/catch(IOE ignored) sequences in cleanup. What it makes easy is to isolate every single operation this way, so if one fails, the next step runs. This makes cleanup potentially more rigorous.
          • S3AUtils list(fs, path, recursive, filter), replicates the classic FileSystem.list(path, filter) but works on the higher performance recursive listFileStatus operation, and returns a list of those. Some bits of code taking List<FileStatus> have had to be changed to List<? extends FileStatus> because despite Java type erasure means this is the same at runtime, it still complains needlessly in the compile.

          Testing

          • Huge magic commit test is called ITestS3AHugeMagicCommits; guarantees excluded with the normal rules. This is important as the test_010_ test case, which creates the file, skips the operation if it runs in parallel...which is going to fail the test with the exact stack trace which aaron saw. That is, I think the test failure really could have been a false alarm. But it forced me to look at cleanup, which is a good thing.
          • New test to create two parallel jobs and work each one simultaneously. Found the listMultipartUploads-prefix bug.
          • The mock tests got very confused; the mock AmazonS3Client now has to track outstanding MPU requests so that listMultipartUploads() can return the full list of active requests, and abortMultipartUpload() raises 404 if there isn't one outstanding. All the little mock answers are now java-8 lambda expressions to make a little bit smaller. Reinstated a comment out bit of assertValidUpload which checked expected tag list vs. actual; I must have commented it out a while back.

          Test run

          Test runs: Endpoint S3 ireland with ddb on

          One transient failure where my explicitly enabled throttling failed too often for the test; Cranked back the normal throttle there from 50% to 25%.

          Tests run: 16, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 101.795 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
          testCommitEmptyFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperations)  Time elapsed: 12.755 sec  <<< ERROR!
          org.apache.hadoop.fs.s3a.AWSServiceThrottledException: Completing multipart commit on fork-0007/test/DELAY_LISTING_ME/testCommitEmptyFile/empty-commit.txt: com.amazonaws.AmazonServiceException: throttled count = 3 (Service: null; Status Code: 503; Error Code: null; Request ID: null):null: throttled count = 3 (Service: null; Status Code: 503; Error Code: null; Request ID: null)
          	at org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client.maybeFail(InconsistentAmazonS3Client.java:611)
          	at org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client.completeMultipartUpload(InconsistentAmazonS3Client.java:537)
          	at org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$finalizeMultipartUpload$1(WriteOperationHelper.java:226)
          	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:108)
          	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:259)
          	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
          	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255)
          	at org.apache.hadoop.fs.s3a.WriteOperationHelper.finalizeMultipartUpload(WriteOperationHelper.java:219)
          	at org.apache.hadoop.fs.s3a.WriteOperationHelper.completeMPUwithRetries(WriteOperationHelper.java:264)
          	at org.apache.hadoop.fs.s3a.commit.CommitOperations.innerCommit(CommitOperations.java:166)
          	at org.apache.hadoop.fs.s3a.commit.CommitOperations.commit(CommitOperations.java:138)
          	at org.apache.hadoop.fs.s3a.commit.CommitOperations.commitOrFail(CommitOperations.java:121)
          	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.commit(ITestCommitOperations.java:325)
          	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.commit(ITestCommitOperations.java:302)
          	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.createCommitAndVerify(ITestCommitOperations.java:284)
          	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.testCommitEmptyFile(ITestCommitOperations.java:202)
          
          Show
          stevel@apache.org Steve Loughran added a comment - Aarons test failure has forced me to look at all the abort code in the committers. So I have. Found bug in the existing S3AFileSystem.listPendingUploads(path), in that as path never had a trailing "/", it'd list pending uploads in any directory which began with the same prefix, e.g. /work/job1 and work/job12. The final committer cleanup() logic would therefore abort adjacent work. Fixed. turned off that bulk cleanup to see what would happen, helped explore that list and delete pending code, which ultimately proved insufficent (see below). listPendingUploads-> abort is in fact the only way to do it reliably. logging improved @debug level, with a bit more on impoort to deal with the use case "things aren't working in production, here are the logs, VM is gone" Two little issues related to cleanup Magic App attempt abort would list all in app attempt dest/_ magic/$jobid $attemptid/{*.pendingset, *.pending} and clean up But this doesn't cleanup on any failed attempt, which will have a different attemptID, so not be found. For the magic committer, switched to a much simpler cleanup of list all uploads pending under $dest, abort magic: delete __magic/* staging: abort wrapped job staging: delete local staging dirs for entire job That is: no list/read/cancel of pending job & task events. Expensive and not complete . This bulk abort of all pending uploads of a dest handles: failed tasks, failed attempts, failure of incomplete jobs from other processes/work which was never aborted. (e.g: a previous query somewhere failed leaving pending work). This was already being done after all the "structured" aborts; here I remove that work and only do the bulk list & rm for a simpler life; made sure all of that excutes in parallel for fastest cleanup. This significantly simplifies the abort, commit failure and helper classes (no need to suppress exceptions), and reduces the diff between the staging & magic committers (so more has been pulled up to the base class) That is: this is the first patch for a while which cuts out production code in any measurable way. The staging committer currently does the same, but it retains the listing and abort of all outstanding requests first. Why? I've kept it there to handle the use case of >1 partitioned commit running simultaneously, where you don't want to abort all the outstanding stuff. That's not fully rounded off in this patch because it's still doing the bulk cleanup: I want to see what Ryan thinks is best here. I think I'm going to opt for: list then optional bulk abort. That means the default doesn't leave pending objects around. That makes the list operation superfluous, of course, but (a) it's going against HDFS so is less expensive and (b) means that all tests cover the codepath, rather than having a condition which has weak coverage. That is: I don't think it's that harmful: there's the cost of an HDFS list(recursive=true), an open + parse of every read; For the magic committer, in constrast, its more expensive (the LIST, the GET) and it doesn't support that notion of partitioned work: you can't get >1 job writing to the same directory tree. Docs: mention examining the _SUCCESS file in troubleshooting section defaults of the various options I'm wrapping up some details on cleanup in the committer arch doc, not included in this patch as it is late on a friday evening. Misc Invoke adds new method ignoreIOExceptions(Log, text, path, lambda-operation) which runs the operation & logs & info if there is a problem. This replaces a lot of the try/catch(IOE ignored) sequences in cleanup. What it makes easy is to isolate every single operation this way, so if one fails, the next step runs. This makes cleanup potentially more rigorous. S3AUtils list(fs, path, recursive, filter) , replicates the classic FileSystem.list(path, filter) but works on the higher performance recursive listFileStatus operation, and returns a list of those. Some bits of code taking List<FileStatus> have had to be changed to List<? extends FileStatus> because despite Java type erasure means this is the same at runtime, it still complains needlessly in the compile. Testing Huge magic commit test is called ITestS3AHugeMagicCommits; guarantees excluded with the normal rules. This is important as the test_010_ test case, which creates the file, skips the operation if it runs in parallel...which is going to fail the test with the exact stack trace which aaron saw . That is, I think the test failure really could have been a false alarm. But it forced me to look at cleanup, which is a good thing. New test to create two parallel jobs and work each one simultaneously. Found the listMultipartUploads-prefix bug. The mock tests got very confused; the mock AmazonS3Client now has to track outstanding MPU requests so that listMultipartUploads() can return the full list of active requests, and abortMultipartUpload() raises 404 if there isn't one outstanding. All the little mock answers are now java-8 lambda expressions to make a little bit smaller. Reinstated a comment out bit of assertValidUpload which checked expected tag list vs. actual; I must have commented it out a while back. Test run Test runs: Endpoint S3 ireland with ddb on One transient failure where my explicitly enabled throttling failed too often for the test; Cranked back the normal throttle there from 50% to 25%. Tests run: 16, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 101.795 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations testCommitEmptyFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperations) Time elapsed: 12.755 sec <<< ERROR! org.apache.hadoop.fs.s3a.AWSServiceThrottledException: Completing multipart commit on fork-0007/test/DELAY_LISTING_ME/testCommitEmptyFile/empty-commit.txt: com.amazonaws.AmazonServiceException: throttled count = 3 (Service: null ; Status Code: 503; Error Code: null ; Request ID: null ): null : throttled count = 3 (Service: null ; Status Code: 503; Error Code: null ; Request ID: null ) at org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client.maybeFail(InconsistentAmazonS3Client.java:611) at org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client.completeMultipartUpload(InconsistentAmazonS3Client.java:537) at org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$finalizeMultipartUpload$1(WriteOperationHelper.java:226) at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:108) at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:259) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313) at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255) at org.apache.hadoop.fs.s3a.WriteOperationHelper.finalizeMultipartUpload(WriteOperationHelper.java:219) at org.apache.hadoop.fs.s3a.WriteOperationHelper.completeMPUwithRetries(WriteOperationHelper.java:264) at org.apache.hadoop.fs.s3a.commit.CommitOperations.innerCommit(CommitOperations.java:166) at org.apache.hadoop.fs.s3a.commit.CommitOperations.commit(CommitOperations.java:138) at org.apache.hadoop.fs.s3a.commit.CommitOperations.commitOrFail(CommitOperations.java:121) at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.commit(ITestCommitOperations.java:325) at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.commit(ITestCommitOperations.java:302) at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.createCommitAndVerify(ITestCommitOperations.java:284) at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.testCommitEmptyFile(ITestCommitOperations.java:202)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ryan: there's always been an option to opt out of success markers; "mapreduce.fileoutputcommitter.marksuccessfuljobs";, default = true. The intermediate map jobs never create them, after all.

          I do like having the marker there though, as the json file is great for testing. Lists the committer used, storage stats off the dest FS, and the list of files created. Makes it trivial to assert that the right committer was used (length == 0 => FileOutputCommitter), and that the right #of files were created.

          I'll not worry about deletion for now though, and will document the property name

          Show
          stevel@apache.org Steve Loughran added a comment - ryan: there's always been an option to opt out of success markers; "mapreduce.fileoutputcommitter.marksuccessfuljobs"; , default = true. The intermediate map jobs never create them, after all. I do like having the marker there though, as the json file is great for testing. Lists the committer used, storage stats off the dest FS, and the list of files created. Makes it trivial to assert that the right committer was used (length == 0 => FileOutputCommitter), and that the right #of files were created. I'll not worry about deletion for now though, and will document the property name
          Hide
          fabbri Aaron Fabbri added a comment -

          Ran through the parallel integration tests about 5 times today without a reproduction. Heisenbug. Wrote a script to run it over and over and save logs overnight. Will shout if I find anything in the morning. Friday we are off on holiday FYI.

          Show
          fabbri Aaron Fabbri added a comment - Ran through the parallel integration tests about 5 times today without a reproduction. Heisenbug. Wrote a script to run it over and over and save logs overnight. Will shout if I find anything in the morning. Friday we are off on holiday FYI.
          Hide
          fabbri Aaron Fabbri added a comment -

          Interesting. Sorry I didn't have more useful comments last night. I added more debug logging and since then have been unable to reproduce it. In bed last night I had an idea that maybe the huge file outputstream was not being closed reliably (thus no .pending being written in aboutToComplete() on close). This morning I looked at the test code {{test_010_CreateHugeFile() }} and it seems to be fine though; it explicitly closes the stream but also handles error case with try-with-resources block. I'll keep trying to reproduce it.

          Show
          fabbri Aaron Fabbri added a comment - Interesting. Sorry I didn't have more useful comments last night. I added more debug logging and since then have been unable to reproduce it. In bed last night I had an idea that maybe the huge file outputstream was not being closed reliably (thus no .pending being written in aboutToComplete() on close). This morning I looked at the test code {{test_010_CreateHugeFile() }} and it seems to be fine though; it explicitly closes the stream but also handles error case with try-with-resources block. I'll keep trying to reproduce it.
          Hide
          rdblue Ryan Blue added a comment -

          I don't think the partitioned committer should continue the _SUCCESS marker convention. Nothing that writes partitioned data currently depends on _SUCCESS markers, so it's easy to avoid the problem entirely because the markers are unreliable: what happens when you're appending data to a partition?

          We implemented a property that allows users to opt in to have _SUCCESS created for the directory output committer only. It creates the _SUCCESS marker after all other operations have finished because that's when we can guarantee that the write was successful. It doesn't delete other markers because there are no well-defined semantics for _SUCCESS with overwrite.

          Show
          rdblue Ryan Blue added a comment - I don't think the partitioned committer should continue the _SUCCESS marker convention. Nothing that writes partitioned data currently depends on _SUCCESS markers, so it's easy to avoid the problem entirely because the markers are unreliable: what happens when you're appending data to a partition? We implemented a property that allows users to opt in to have _SUCCESS created for the directory output committer only. It creates the _SUCCESS marker after all other operations have finished because that's when we can guarantee that the write was successful. It doesn't delete other markers because there are no well-defined semantics for _SUCCESS with overwrite.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Think I've found one little issue which could create problems, though not by its own the root cause. Seem closer to replicating it though, and yes, seems related to cleanup of jobs.

          One little feature is that the committers were being (over?) zealous in aborting all MPUs under their destination path, on the basis that failed tasks could have left outstanding MPUs which, if the data were not persisted, would not be enumerable by looking for .pendingset files.

          But S3aFileSystem.listMultipartUploads(prefix) actually turns out to list everything matching the prefix, even in parallel dirs, as it isn't adding a "/" suffix to say "directory only". Which meant if you had a job commit to "test/myjob", i'd also delete pending uploads to "test/myjob2"

          There's more to it than that; I think I'll need to review all listing stuff to be sure, but now I have some tests failing. Provided the tests themselves are correct, then I'll be able to find/fix it.

          Show
          stevel@apache.org Steve Loughran added a comment - Think I've found one little issue which could create problems, though not by its own the root cause. Seem closer to replicating it though, and yes, seems related to cleanup of jobs. One little feature is that the committers were being (over?) zealous in aborting all MPUs under their destination path, on the basis that failed tasks could have left outstanding MPUs which, if the data were not persisted, would not be enumerable by looking for .pendingset files. But S3aFileSystem.listMultipartUploads(prefix) actually turns out to list everything matching the prefix, even in parallel dirs, as it isn't adding a "/" suffix to say "directory only". Which meant if you had a job commit to "test/myjob", i'd also delete pending uploads to "test/myjob2" There's more to it than that; I think I'll need to review all listing stuff to be sure, but now I have some tests failing. Provided the tests themselves are correct, then I'll be able to find/fix it.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Side issue, one for Ryan Blue in particular

          If the job commit fails, should any existing _SUCCESS file be deleted? Because things may be inconsistent & we shouldn't have any cue in the dir that it contains valid data.

          What I'd do is: delete the marker before any files are committed or deleted, but after any conflict resolution checks which may fail the job have run. That way, if a job commit is aborted: the existing marker is unchanged.

          • directory committer: delete it for the APPEND case; retain for FAIL (its deleted for free in OVERWRITE)
          • partitioned committer: both OVERWRITE and APPEND to delete the marker, though append will

          Moot for the magic commit as it doesn't do any in-situ overwrites/appends of data.

          If you think we should do this, I can add it as a folowup JIRA; would need test modification &c

          Show
          stevel@apache.org Steve Loughran added a comment - Side issue, one for Ryan Blue in particular If the job commit fails, should any existing _SUCCESS file be deleted? Because things may be inconsistent & we shouldn't have any cue in the dir that it contains valid data. What I'd do is: delete the marker before any files are committed or deleted, but after any conflict resolution checks which may fail the job have run. That way, if a job commit is aborted: the existing marker is unchanged. directory committer: delete it for the APPEND case; retain for FAIL (its deleted for free in OVERWRITE) partitioned committer: both OVERWRITE and APPEND to delete the marker, though append will Moot for the magic commit as it doesn't do any in-situ overwrites/appends of data. If you think we should do this, I can add it as a folowup JIRA; would need test modification &c
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks for this, I'll look through the @after code too.

          1. All the other scale tests work in parallel, so I don't think it's their teardown, more something happening in a parallel test
          2. which can include a job commit happening on a different path
          3. as well as some test setup doing autopurge of old stuff
          4. test is getting filename inconsistent; not correctly working out path of pending file

          The fact that you can see it and I can't makes me think it's either some race condition or config. The fact you can consistently see it means that it's unlikely to be a race, as that's going to to fail intermittently. I'd consider parallel tests purging buckets. But: how is that going to clean up the file under __magic? Lose the uncommitted MPUs, yes.

          Here's my plan

          1. make sure that the code logs @ info when a magic file is saved, some message like "file $dest... ready to commit, commit metadata stored at $pending."
          2. failing test case to add some asserts in the test about existence of pending files under a path. This will help differentiate purge of commit data from purge of files. Adding it in the test case which creates the file will verify that any delete/purge happens between the two
          3. add a test which explicitly deletes the MPU of a pending commit, see what happens
          4. add a test which explicitly runs two job commits in parallel to separate paths: verifies isolation.
          5. review troubleshooting docs. Once your list MPU CI
          6. Have all commands to purge pending MPUs actually list@info the files. After all, in normal execution there shouldn't be any.
            That CLI you've proposed for listing MPUS, "hadoop s3guard uploads" will help diagnose stuff in the field
          Show
          stevel@apache.org Steve Loughran added a comment - thanks for this, I'll look through the @after code too. All the other scale tests work in parallel, so I don't think it's their teardown, more something happening in a parallel test which can include a job commit happening on a different path as well as some test setup doing autopurge of old stuff test is getting filename inconsistent; not correctly working out path of pending file The fact that you can see it and I can't makes me think it's either some race condition or config. The fact you can consistently see it means that it's unlikely to be a race, as that's going to to fail intermittently. I'd consider parallel tests purging buckets. But: how is that going to clean up the file under __magic? Lose the uncommitted MPUs, yes. Here's my plan make sure that the code logs @ info when a magic file is saved, some message like "file $dest... ready to commit, commit metadata stored at $pending." failing test case to add some asserts in the test about existence of pending files under a path. This will help differentiate purge of commit data from purge of files. Adding it in the test case which creates the file will verify that any delete/purge happens between the two add a test which explicitly deletes the MPU of a pending commit, see what happens add a test which explicitly runs two job commits in parallel to separate paths: verifies isolation. review troubleshooting docs. Once your list MPU CI Have all commands to purge pending MPUs actually list@info the files. After all, in normal execution there shouldn't be any. That CLI you've proposed for listing MPUS, "hadoop s3guard uploads" will help diagnose stuff in the field
          Hide
          fabbri Aaron Fabbri added a comment - - edited

          That's not a minor failure though

          I was assuming this is a test code failure, not logic issues in core S3A or even in the "experimental" magic committer codepath, based on the rest of the testing I've done.

          (edit: Looks like the @After deletes /test, not /tests3aScale. So much for that theory. Shortening this a bit)

          Curious this test case ITestHugeMagicCommits always succeeds unless I turn on parallel tests. It looks like your excluding the test from parallel execution.

          <exclude>**/ITestS3AHuge*.java</exclude>
          <exclude>**/ITestHuge*.java</exclude>
          

          The ITestHugeMagicCommits thing is a complex way to get this test coverage of S3A's handling of magic paths on create. I spent some time earlier reading through it.

          Though we depend on the ordering of test cases to make them stateful (understood considering the overhead associated with setting up the huge files), we still have @Before and @After functions which do a bunch of stuff between the ordered test cases. Things happening in the four layers of @Before overrides, from super to sub class, roughly:

          • creating fs contract
          • getting test filesystem from the contract Couldn't this purge multiparts and break this test case? I guess if someone sets a very low age value in their config.
          • mkdirs(testPath)
          • testPath = path("/tests3ascale") in S3AScaleTestBase.. which defines getter getTestPath()
          • assume scale tests are enabled
          • blah blah
          • set up some paths for the __magic directory in ITestHugeMagicCommits

          Now after each test case, looking at @After functions:

          • Delete test dir, recursively. Looking at logs, deletes /test, not /tests3ascale, though.
          • fs.close()

          Spending a little time looking at debug logs after dinner I'll shout if I get more detail.

          Show
          fabbri Aaron Fabbri added a comment - - edited That's not a minor failure though I was assuming this is a test code failure, not logic issues in core S3A or even in the "experimental" magic committer codepath, based on the rest of the testing I've done. (edit: Looks like the @After deletes /test, not /tests3aScale. So much for that theory. Shortening this a bit) Curious this test case ITestHugeMagicCommits always succeeds unless I turn on parallel tests. It looks like your excluding the test from parallel execution. <exclude>**/ITestS3AHuge*.java</exclude> <exclude>**/ITestHuge*.java</exclude> The ITestHugeMagicCommits thing is a complex way to get this test coverage of S3A's handling of magic paths on create. I spent some time earlier reading through it. Though we depend on the ordering of test cases to make them stateful (understood considering the overhead associated with setting up the huge files), we still have @Before and @After functions which do a bunch of stuff between the ordered test cases. Things happening in the four layers of @Before overrides, from super to sub class, roughly: creating fs contract getting test filesystem from the contract Couldn't this purge multiparts and break this test case? I guess if someone sets a very low age value in their config. mkdirs(testPath) testPath = path("/tests3ascale") in S3AScaleTestBase.. which defines getter getTestPath() assume scale tests are enabled blah blah set up some paths for the __magic directory in ITestHugeMagicCommits Now after each test case, looking at @After functions: Delete test dir , recursively. Looking at logs, deletes /test, not /tests3ascale, though. fs.close() Spending a little time looking at debug logs after dinner I'll shout if I get more detail.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          That's not a minor failure though, is it: it's saying the metadata file listing everything to commit is missing. Let me look at it tomorrow

          w.r.t checkstyle, I'd left the ones in S3GuardTool and DynamoDBMetadataStore alone as I felt they were unrelated

          Show
          stevel@apache.org Steve Loughran added a comment - That's not a minor failure though, is it: it's saying the metadata file listing everything to commit is missing. Let me look at it tomorrow w.r.t checkstyle, I'd left the ones in S3GuardTool and DynamoDBMetadataStore alone as I felt they were unrelated
          Hide
          fabbri Aaron Fabbri added a comment -

          Latest changes look OK.

          javac warnings are deprecation (unavoidable AFAICT). It looks like a couple of the checkstyle warnings can be fixed still. Many are unavoidable (parameter counts, javadoc link line lengths).

          Whitespace needs to be fixed when applying patch, of course.

          I still reproduce a failure with parallel tests here:

          Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 31.163 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.magic.ITestHugeMagicCommits
          test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.commit.magic.ITestHugeMagicCommits)  Time elapsed: 1.551 sec  <<< ERROR!
          java.io.FileNotFoundException: No pending file: not found s3a://fabbri-new/tests3ascale/ITestHugeMagicCommits/commit/__magic/job_001/commit.bin.pending in s3a://fabbri-new/tests3ascale/ITestHugeMagicCommits/commit/__magic/job_001
                  at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2205)
                  at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2099)
                  at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2038)
                  at org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:900)
                  at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:884)
                  at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294)
                  at org.apache.hadoop.fs.s3a.commit.magic.ITestHugeMagicCommits.test_030_postCreationAssertions(ITestHugeMagicCommits.java:125)
          

          Which went away when I ran the test in isolation. Again don't consider this a blocker as (1) only affects parallel (2) magic not staging, (3) we need to get this stuff committed to a branch so others can collaborate on polish like parallel test flakiness etc.

          I'll let you fix the checkstyle since you asked me not to help with committing it.

          Show
          fabbri Aaron Fabbri added a comment - Latest changes look OK. javac warnings are deprecation (unavoidable AFAICT). It looks like a couple of the checkstyle warnings can be fixed still. Many are unavoidable (parameter counts, javadoc link line lengths). Whitespace needs to be fixed when applying patch, of course. I still reproduce a failure with parallel tests here: Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 31.163 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.magic.ITestHugeMagicCommits test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.commit.magic.ITestHugeMagicCommits) Time elapsed: 1.551 sec <<< ERROR! java.io.FileNotFoundException: No pending file: not found s3a://fabbri-new/tests3ascale/ITestHugeMagicCommits/commit/__magic/job_001/commit.bin.pending in s3a://fabbri-new/tests3ascale/ITestHugeMagicCommits/commit/__magic/job_001 at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2205) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2099) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2038) at org.apache.hadoop.fs.contract.ContractTestUtils.verifyPathExists(ContractTestUtils.java:900) at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathExists(ContractTestUtils.java:884) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertPathExists(AbstractFSContractTestBase.java:294) at org.apache.hadoop.fs.s3a.commit.magic.ITestHugeMagicCommits.test_030_postCreationAssertions(ITestHugeMagicCommits.java:125) Which went away when I ran the test in isolation. Again don't consider this a blocker as (1) only affects parallel (2) magic not staging, (3) we need to get this stuff committed to a branch so others can collaborate on polish like parallel test flakiness etc. I'll let you fix the checkstyle since you asked me not to help with committing it.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 9m 48s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 17m 2s trunk passed
          +1 compile 12m 44s trunk passed
          +1 checkstyle 2m 6s trunk passed
          +1 mvnsite 3m 21s trunk passed
          +1 shadedclient 15m 32s branch has no errors when building and testing our client artifacts.
          +1 findbugs 5m 0s trunk passed
          +1 javadoc 2m 33s trunk passed
                Patch Compile Tests
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 3m 0s the patch passed
          +1 compile 13m 49s the patch passed
          -1 javac 13m 49s root generated 2 new + 1240 unchanged - 0 fixed = 1242 total (was 1240)
          -0 checkstyle 2m 12s root: The patch generated 29 new + 116 unchanged - 22 fixed = 145 total (was 138)
          +1 mvnsite 4m 5s the patch passed
          -1 whitespace 0m 0s The patch has 58 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 1s The patch 3 line(s) with tabs.
          +1 xml 0m 7s The patch has no ill-formed XML file.
          +1 shadedclient 9m 50s patch has no errors when building and testing our client artifacts.
          +1 findbugs 5m 4s the patch passed
          +1 javadoc 2m 33s the patch passed
                Other Tests
          +1 unit 7m 32s hadoop-common in the patch passed.
          +1 unit 0m 54s hadoop-yarn-registry in the patch passed.
          +1 unit 3m 1s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 9m 10s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 4m 41s hadoop-aws in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          132m 51s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896465/HADOOP-13786-046.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 0371ebf7e5ae 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 13fa2d4
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/artifact/out/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/artifact/out/whitespace-tabs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/testReport/
          Max. process+thread count 1391 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 9m 48s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 17m 2s trunk passed +1 compile 12m 44s trunk passed +1 checkstyle 2m 6s trunk passed +1 mvnsite 3m 21s trunk passed +1 shadedclient 15m 32s branch has no errors when building and testing our client artifacts. +1 findbugs 5m 0s trunk passed +1 javadoc 2m 33s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 3m 0s the patch passed +1 compile 13m 49s the patch passed -1 javac 13m 49s root generated 2 new + 1240 unchanged - 0 fixed = 1242 total (was 1240) -0 checkstyle 2m 12s root: The patch generated 29 new + 116 unchanged - 22 fixed = 145 total (was 138) +1 mvnsite 4m 5s the patch passed -1 whitespace 0m 0s The patch has 58 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 1s The patch 3 line(s) with tabs. +1 xml 0m 7s The patch has no ill-formed XML file. +1 shadedclient 9m 50s patch has no errors when building and testing our client artifacts. +1 findbugs 5m 4s the patch passed +1 javadoc 2m 33s the patch passed       Other Tests +1 unit 7m 32s hadoop-common in the patch passed. +1 unit 0m 54s hadoop-yarn-registry in the patch passed. +1 unit 3m 1s hadoop-mapreduce-client-core in the patch passed. +1 unit 9m 10s hadoop-mapreduce-client-app in the patch passed. +1 unit 4m 41s hadoop-aws in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 132m 51s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896465/HADOOP-13786-046.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 0371ebf7e5ae 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 13fa2d4 maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/artifact/out/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/artifact/out/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/testReport/ Max. process+thread count 1391 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13640/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          If you give the vote, I'll do the merge: I'll want to do some final checks against the latest trunk version. Also, I'd like to be able to tick it off as done

          Show
          stevel@apache.org Steve Loughran added a comment - If you give the vote, I'll do the merge: I'll want to do some final checks against the latest trunk version. Also, I'd like to be able to tick it off as done
          Hide
          fabbri Aaron Fabbri added a comment -

          I agree trunk is the right place for this now that we've spent a lot of time testing and reviewing. I will review the latest changes and do another test run. If everything looks good I will commit it.

          Show
          fabbri Aaron Fabbri added a comment - I agree trunk is the right place for this now that we've spent a lot of time testing and reviewing. I will review the latest changes and do another test run. If everything looks good I will commit it.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 046, tune tests + two spaces of indentation in the production code (checkstyle)

          huge file IO
          *ITestHugeMagicCommits skips test_040_PositionedReadHugeFile.

          • All subclasses generate unique pathname for test, even under forked VM path
          • added final "test case" which prints the StorageStatus for S3A (as this is shared across instances, it gives the aggregate across the entire VM)

          Mock IO

          • override the S3A operationRetried() method. This had been triggering NPEs which were being (correctly) swallowed in the retry logic...but they shouldn't have been raised at all. fixed.
          • Use the word "mock" in all mock exceptions raised, to make clear they're mock exceptions

          Other

          • cut ref to obsolete JetS3t test from pom.xml
          • indent two spaces of FileOutputFormat

          Aaron: I'm done here; there's nothing left to do in this patch. Code done, tests done, docs done, downstream integration done. It's ready to go.
          I don't think a separate branch will add much. Putting it into trunk and letting other people play with it and its features (especially retry) is the way to finish it off. I don't think I'll break anything, but a merge to trunk is the only way we will find this out. Also: makes it easier for anyone to build & test the spark cloud-integration module, as they don't need to build hadoop 3.1-snapshot. Again: gets broader use

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 046, tune tests + two spaces of indentation in the production code (checkstyle) huge file IO * ITestHugeMagicCommits skips test_040_PositionedReadHugeFile . All subclasses generate unique pathname for test, even under forked VM path added final "test case" which prints the StorageStatus for S3A (as this is shared across instances, it gives the aggregate across the entire VM) Mock IO override the S3A operationRetried() method. This had been triggering NPEs which were being (correctly) swallowed in the retry logic...but they shouldn't have been raised at all. fixed. Use the word "mock" in all mock exceptions raised, to make clear they're mock exceptions Other cut ref to obsolete JetS3t test from pom.xml indent two spaces of FileOutputFormat Aaron: I'm done here; there's nothing left to do in this patch. Code done, tests done, docs done, downstream integration done. It's ready to go. I don't think a separate branch will add much. Putting it into trunk and letting other people play with it and its features (especially retry) is the way to finish it off. I don't think I'll break anything, but a merge to trunk is the only way we will find this out. Also: makes it easier for anyone to build & test the spark cloud-integration module, as they don't need to build hadoop 3.1-snapshot. Again: gets broader use
          Hide
          stevel@apache.org Steve Loughran added a comment -

          that huge read test is the one designed to execute each method in a strict order (which is why we have BASIC/APL-style numbering of each methods. It does still use the fork ID to calculate the test path. And they should all be set up to run in the sequential phase anyway

          Looking at the new scale test ITestHugeMagicCommits, its overridden the tests (50, 100) to skip them as superfluous, and test 999, the cleanup adding
          an abort for all commits as well as an extra delete of the magic dir (which should again be superfluous).

          Also, even though the tests are using fork-specific paths, they are using the same path in each of the huge tests for their own files; I like to isolate them more just for all that metadata consistency stuff. So I'll set the logic for all the huge reads to do that

          1. I'll mark the 040 test as unneeded there too
          2. review the bits of the pom where the test sequences are set up. Looks like there's an old ref to a JetS3t test there to go.
          3. And review the test _999 teardown
          Show
          stevel@apache.org Steve Loughran added a comment - that huge read test is the one designed to execute each method in a strict order (which is why we have BASIC/APL-style numbering of each methods. It does still use the fork ID to calculate the test path. And they should all be set up to run in the sequential phase anyway Looking at the new scale test ITestHugeMagicCommits , its overridden the tests (50, 100) to skip them as superfluous, and test 999, the cleanup adding an abort for all commits as well as an extra delete of the magic dir (which should again be superfluous). Also, even though the tests are using fork-specific paths, they are using the same path in each of the huge tests for their own files; I like to isolate them more just for all that metadata consistency stuff. So I'll set the logic for all the huge reads to do that I'll mark the 040 test as unneeded there too review the bits of the pom where the test sequences are set up. Looks like there's an old ref to a JetS3t test there to go. And review the test _999 teardown
          Hide
          fabbri Aaron Fabbri added a comment -

          Ok I reviewed all the newest changes on the pull request
          (as of writing, most recent commit is df094d7).

          I applied the patch to trunk and ran all the S3A integration tests in us-west-2. I had some failures in the "huge" magic committer test:

            ITestHugeMagicCommits.test_030_postCreationAssertions:112->AbstractFSContractTestBase.assertPathExists:294 » FileNotFound
            ITestHugeMagicCommits>AbstractSTestS3AHugeFiles.test_040_PositionedReadHugeFile:351->AbstractSTestS3AHugeFiles.assumeHugeFileExists:310->AbstractSTestS3AHugeFiles.assumeFileExists:320 » FileNotFound
          

          which went away when I ran tests in serial mode. Minus that, and considering that the staging committer is the initial focus, this feels like it is ready for a merge. Do you want to put this on a feature branch (short term only please) or commit to trunk?

          Show
          fabbri Aaron Fabbri added a comment - Ok I reviewed all the newest changes on the pull request (as of writing, most recent commit is df094d7). I applied the patch to trunk and ran all the S3A integration tests in us-west-2. I had some failures in the "huge" magic committer test: ITestHugeMagicCommits.test_030_postCreationAssertions:112->AbstractFSContractTestBase.assertPathExists:294 » FileNotFound ITestHugeMagicCommits>AbstractSTestS3AHugeFiles.test_040_PositionedReadHugeFile:351->AbstractSTestS3AHugeFiles.assumeHugeFileExists:310->AbstractSTestS3AHugeFiles.assumeFileExists:320 » FileNotFound which went away when I ran tests in serial mode. Minus that, and considering that the staging committer is the initial focus, this feels like it is ready for a merge. Do you want to put this on a feature branch (short term only please) or commit to trunk?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 2s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 46s Maven dependency ordering for branch
          +1 mvninstall 24m 29s trunk passed
          +1 compile 18m 14s trunk passed
          +1 checkstyle 2m 55s trunk passed
          +1 mvnsite 5m 47s trunk passed
          +1 shadedclient 21m 18s branch has no errors when building and testing our client artifacts.
          +1 findbugs 7m 10s trunk passed
          +1 javadoc 3m 44s trunk passed
                Patch Compile Tests
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 3m 44s the patch passed
          +1 compile 15m 29s the patch passed
          -1 javac 15m 29s root generated 2 new + 1238 unchanged - 0 fixed = 1240 total (was 1238)
          -0 checkstyle 2m 34s root: The patch generated 27 new + 118 unchanged - 21 fixed = 145 total (was 139)
          +1 mvnsite 5m 36s the patch passed
          -1 whitespace 0m 0s The patch has 58 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 3s The patch 3 line(s) with tabs.
          +1 xml 0m 11s The patch has no ill-formed XML file.
          +1 shadedclient 12m 58s patch has no errors when building and testing our client artifacts.
          +1 findbugs 7m 29s the patch passed
          +1 javadoc 3m 56s the patch passed
                Other Tests
          +1 unit 11m 50s hadoop-common in the patch passed.
          +1 unit 1m 16s hadoop-yarn-registry in the patch passed.
          +1 unit 4m 21s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 11m 44s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 2m 14s hadoop-aws in the patch passed.
          +1 asflicense 0m 45s The patch does not generate ASF License warnings.
          176m 0s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896219/HADOOP-13786-045.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 6faf5349d2d7 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / dcd99c4
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/artifact/out/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/artifact/out/whitespace-tabs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/testReport/
          Max. process+thread count 1411 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 2s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 1m 46s Maven dependency ordering for branch +1 mvninstall 24m 29s trunk passed +1 compile 18m 14s trunk passed +1 checkstyle 2m 55s trunk passed +1 mvnsite 5m 47s trunk passed +1 shadedclient 21m 18s branch has no errors when building and testing our client artifacts. +1 findbugs 7m 10s trunk passed +1 javadoc 3m 44s trunk passed       Patch Compile Tests 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 3m 44s the patch passed +1 compile 15m 29s the patch passed -1 javac 15m 29s root generated 2 new + 1238 unchanged - 0 fixed = 1240 total (was 1238) -0 checkstyle 2m 34s root: The patch generated 27 new + 118 unchanged - 21 fixed = 145 total (was 139) +1 mvnsite 5m 36s the patch passed -1 whitespace 0m 0s The patch has 58 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 3s The patch 3 line(s) with tabs. +1 xml 0m 11s The patch has no ill-formed XML file. +1 shadedclient 12m 58s patch has no errors when building and testing our client artifacts. +1 findbugs 7m 29s the patch passed +1 javadoc 3m 56s the patch passed       Other Tests +1 unit 11m 50s hadoop-common in the patch passed. +1 unit 1m 16s hadoop-yarn-registry in the patch passed. +1 unit 4m 21s hadoop-mapreduce-client-core in the patch passed. +1 unit 11m 44s hadoop-mapreduce-client-app in the patch passed. +1 unit 2m 14s hadoop-aws in the patch passed. +1 asflicense 0m 45s The patch does not generate ASF License warnings. 176m 0s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896219/HADOOP-13786-045.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 6faf5349d2d7 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / dcd99c4 maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/artifact/out/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/artifact/out/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/testReport/ Max. process+thread count 1411 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13634/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          yes, just pushed it out. not rebased for a while either

          Show
          stevel@apache.org Steve Loughran added a comment - yes, just pushed it out. not rebased for a while either
          Hide
          fabbri Aaron Fabbri added a comment -

          Testing / reviewing the latest here. Your pull request is still up to sync with this, right? That way I can just view the delta deltas.

          Show
          fabbri Aaron Fabbri added a comment - Testing / reviewing the latest here. Your pull request is still up to sync with this, right? That way I can just view the delta deltas.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 045

          • fix checkstyle warnings as far as possible (that's pretty much it now)
            *review and edit hadoop-aws site docs; rendering, looking for errors. Fixing any errors. The previous patch had a bit of the docs of HADOOP-14965 in, as I'd applied that patch over for perf and failed to roll that bit back. oops!

          Testing: s3 ireland w/ dynamo; downstream in my cloud integration tests too

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 045 fix checkstyle warnings as far as possible (that's pretty much it now) *review and edit hadoop-aws site docs; rendering, looking for errors. Fixing any errors. The previous patch had a bit of the docs of HADOOP-14965 in, as I'd applied that patch over for perf and failed to roll that bit back. oops! Testing: s3 ireland w/ dynamo; downstream in my cloud integration tests too
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 1s The patch appears to include 58 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 15m 54s trunk passed
          +1 compile 11m 52s trunk passed
          +1 checkstyle 2m 7s trunk passed
          +1 mvnsite 3m 23s trunk passed
          +1 shadedclient 16m 9s branch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 40s trunk passed
          +1 javadoc 2m 36s trunk passed
                Patch Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 38s the patch passed
          +1 compile 12m 11s the patch passed
          -1 javac 12m 11s root generated 2 new + 1238 unchanged - 0 fixed = 1240 total (was 1238)
          -0 checkstyle 2m 12s root: The patch generated 29 new + 117 unchanged - 21 fixed = 146 total (was 138)
          +1 mvnsite 3m 38s the patch passed
          -1 whitespace 0m 0s The patch has 53 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 1s The patch 3 line(s) with tabs.
          +1 xml 0m 7s The patch has no ill-formed XML file.
          +1 shadedclient 10m 6s patch has no errors when building and testing our client artifacts.
          +1 findbugs 5m 42s the patch passed
          +1 javadoc 3m 18s the patch passed
                Other Tests
          +1 unit 8m 45s hadoop-common in the patch passed.
          +1 unit 1m 10s hadoop-yarn-registry in the patch passed.
          +1 unit 3m 22s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 9m 10s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 2m 16s hadoop-aws in the patch passed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          120m 17s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895967/HADOOP-13786-044.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 97a14a28ad47 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4d2dce4
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/artifact/out/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/artifact/out/whitespace-tabs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/testReport/
          Max. process+thread count 1375 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 1s The patch appears to include 58 new or modified test files.       trunk Compile Tests 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 15m 54s trunk passed +1 compile 11m 52s trunk passed +1 checkstyle 2m 7s trunk passed +1 mvnsite 3m 23s trunk passed +1 shadedclient 16m 9s branch has no errors when building and testing our client artifacts. +1 findbugs 4m 40s trunk passed +1 javadoc 2m 36s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 38s the patch passed +1 compile 12m 11s the patch passed -1 javac 12m 11s root generated 2 new + 1238 unchanged - 0 fixed = 1240 total (was 1238) -0 checkstyle 2m 12s root: The patch generated 29 new + 117 unchanged - 21 fixed = 146 total (was 138) +1 mvnsite 3m 38s the patch passed -1 whitespace 0m 0s The patch has 53 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 1s The patch 3 line(s) with tabs. +1 xml 0m 7s The patch has no ill-formed XML file. +1 shadedclient 10m 6s patch has no errors when building and testing our client artifacts. +1 findbugs 5m 42s the patch passed +1 javadoc 3m 18s the patch passed       Other Tests +1 unit 8m 45s hadoop-common in the patch passed. +1 unit 1m 10s hadoop-yarn-registry in the patch passed. +1 unit 3m 22s hadoop-mapreduce-client-core in the patch passed. +1 unit 9m 10s hadoop-mapreduce-client-app in the patch passed. +1 unit 2m 16s hadoop-aws in the patch passed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 120m 17s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895967/HADOOP-13786-044.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 97a14a28ad47 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 4d2dce4 maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/artifact/out/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/artifact/out/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/testReport/ Max. process+thread count 1375 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13625/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 044 switch throttling retry policy to exponential, as it apparently helps S3 recover faster, added text about SSE-KMS and S3Guard/DDB throttling too.

          Tested: s3 ireland without s3guard; all well apart from the failure in ITestS3AFileOperationCost which has a patch pending already

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 044 switch throttling retry policy to exponential, as it apparently helps S3 recover faster, added text about SSE-KMS and S3Guard/DDB throttling too. Tested: s3 ireland without s3guard; all well apart from the failure in ITestS3AFileOperationCost which has a patch pending already
          Hide
          stevel@apache.org Steve Loughran added a comment -

          fix failing TestConfigurationFields; checkstyle & javadocs.

          Fix was done by moving mapreduce.outputcommitter.factory.scheme.s3a to mapred-default, updating TestMapreduceConfigFiles to handle PathOutputCommitterFactory fields and fs-specific schemas

          checkstyle & javadocs are at the limit of what can be done: its complaints about FileOutputFormat & s3guard files are unrelated, and the length ones are about URLs in javadocs & other things which can't be shortened.

          regarding the 443 error, maybe it was 444: I've added both for now with a comment. Given how rare it is, we could pull that case (leave the exception around for future though).

          Show
          stevel@apache.org Steve Loughran added a comment - fix failing TestConfigurationFields; checkstyle & javadocs. Fix was done by moving mapreduce.outputcommitter.factory.scheme.s3a to mapred-default, updating TestMapreduceConfigFiles to handle PathOutputCommitterFactory fields and fs-specific schemas checkstyle & javadocs are at the limit of what can be done: its complaints about FileOutputFormat & s3guard files are unrelated, and the length ones are about URLs in javadocs & other things which can't be shortened. regarding the 443 error, maybe it was 444: I've added both for now with a comment. Given how rare it is, we could pull that case (leave the exception around for future though).
          Hide
          fabbri Aaron Fabbri added a comment -

          Shout if you want help with getting these to be green. I spent some time yesterday on the checkstyle stuff but you posted a patch.. Figured this time I'd check with you first.

          Show
          fabbri Aaron Fabbri added a comment - Shout if you want help with getting these to be green. I spent some time yesterday on the checkstyle stuff but you posted a patch.. Figured this time I'd check with you first.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          TestCommonConfigurationFields failing as with the mapreduce s3a binding...move away from rest of S3A into mapred-default; make sure its config tests are happy

          Show
          stevel@apache.org Steve Loughran added a comment - TestCommonConfigurationFields failing as with the mapreduce s3a binding...move away from rest of S3A into mapred-default; make sure its config tests are happy
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 57 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 46s Maven dependency ordering for branch
          +1 mvninstall 23m 20s trunk passed
          +1 compile 17m 20s trunk passed
          +1 checkstyle 2m 14s trunk passed
          +1 mvnsite 3m 20s trunk passed
          +1 shadedclient 16m 9s branch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 51s trunk passed
          +1 javadoc 2m 22s trunk passed
                Patch Compile Tests
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 2m 51s the patch passed
          +1 compile 14m 47s the patch passed
          -1 javac 14m 47s root generated 2 new + 1238 unchanged - 0 fixed = 1240 total (was 1238)
          -0 checkstyle 2m 9s root: The patch generated 39 new + 117 unchanged - 19 fixed = 156 total (was 136)
          +1 mvnsite 3m 30s the patch passed
          -1 whitespace 0m 0s The patch has 48 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 1s The patch 3 line(s) with tabs.
          +1 xml 0m 7s The patch has no ill-formed XML file.
          +1 shadedclient 9m 54s patch has no errors when building and testing our client artifacts.
          +1 findbugs 5m 33s the patch passed
          -1 javadoc 0m 34s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
          -1 javadoc 0m 29s hadoop-tools_hadoop-aws generated 9 new + 1 unchanged - 0 fixed = 10 total (was 1)
                Other Tests
          -1 unit 10m 19s hadoop-common in the patch failed.
          +1 unit 0m 56s hadoop-yarn-registry in the patch passed.
          +1 unit 3m 27s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 1m 51s hadoop-aws in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          127m 35s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895508/HADOOP-13786-042.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 3458bc71242a 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / b00f828
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/whitespace-tabs.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/testReport/
          Max. process+thread count 1343 (vs. ulimit of 5000)
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 57 new or modified test files.       trunk Compile Tests 0 mvndep 1m 46s Maven dependency ordering for branch +1 mvninstall 23m 20s trunk passed +1 compile 17m 20s trunk passed +1 checkstyle 2m 14s trunk passed +1 mvnsite 3m 20s trunk passed +1 shadedclient 16m 9s branch has no errors when building and testing our client artifacts. +1 findbugs 4m 51s trunk passed +1 javadoc 2m 22s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 51s the patch passed +1 compile 14m 47s the patch passed -1 javac 14m 47s root generated 2 new + 1238 unchanged - 0 fixed = 1240 total (was 1238) -0 checkstyle 2m 9s root: The patch generated 39 new + 117 unchanged - 19 fixed = 156 total (was 136) +1 mvnsite 3m 30s the patch passed -1 whitespace 0m 0s The patch has 48 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 1s The patch 3 line(s) with tabs. +1 xml 0m 7s The patch has no ill-formed XML file. +1 shadedclient 9m 54s patch has no errors when building and testing our client artifacts. +1 findbugs 5m 33s the patch passed -1 javadoc 0m 34s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) -1 javadoc 0m 29s hadoop-tools_hadoop-aws generated 9 new + 1 unchanged - 0 fixed = 10 total (was 1)       Other Tests -1 unit 10m 19s hadoop-common in the patch failed. +1 unit 0m 56s hadoop-yarn-registry in the patch passed. +1 unit 3m 27s hadoop-mapreduce-client-core in the patch passed. +1 unit 1m 51s hadoop-aws in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 127m 35s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895508/HADOOP-13786-042.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 3458bc71242a 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / b00f828 maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/whitespace-tabs.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/testReport/ Max. process+thread count 1343 (vs. ulimit of 5000) modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13621/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 042, checkstyle

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 042, checkstyle
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 57 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 16m 47s trunk passed
          +1 compile 12m 9s trunk passed
          +1 checkstyle 2m 11s trunk passed
          +1 mvnsite 2m 49s trunk passed
          +1 shadedclient 15m 23s branch has no errors when building and testing our client artifacts.
          +1 findbugs 3m 30s trunk passed
          +1 javadoc 2m 1s trunk passed
                Patch Compile Tests
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 56s the patch passed
          +1 compile 11m 53s the patch passed
          -1 javac 11m 53s root generated 2 new + 1247 unchanged - 0 fixed = 1249 total (was 1247)
          -0 checkstyle 2m 18s root: The patch generated 98 new + 117 unchanged - 19 fixed = 215 total (was 136)
          +1 mvnsite 2m 51s the patch passed
          -1 whitespace 0m 0s The patch has 45 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 1s The patch 3 line(s) with tabs.
          +1 xml 0m 7s The patch has no ill-formed XML file.
          +1 shadedclient 10m 24s patch has no errors when building and testing our client artifacts.
          +1 findbugs 4m 36s the patch passed
          -1 javadoc 0m 31s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
          -1 javadoc 0m 23s hadoop-tools_hadoop-aws generated 9 new + 1 unchanged - 0 fixed = 10 total (was 1)
                Other Tests
          -1 unit 8m 49s hadoop-common in the patch failed.
          +1 unit 0m 53s hadoop-yarn-registry in the patch passed.
          +1 unit 3m 9s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 1m 38s hadoop-aws in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          104m 34s



          Reason Tests
          Failed junit tests hadoop.http.TestHttpServer
            hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue HADOOP-15003
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895197/HADOOP-13786-041.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux c5e2667bdeca 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 56b88b0
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/whitespace-tabs.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 57 new or modified test files.       trunk Compile Tests 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 16m 47s trunk passed +1 compile 12m 9s trunk passed +1 checkstyle 2m 11s trunk passed +1 mvnsite 2m 49s trunk passed +1 shadedclient 15m 23s branch has no errors when building and testing our client artifacts. +1 findbugs 3m 30s trunk passed +1 javadoc 2m 1s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 56s the patch passed +1 compile 11m 53s the patch passed -1 javac 11m 53s root generated 2 new + 1247 unchanged - 0 fixed = 1249 total (was 1247) -0 checkstyle 2m 18s root: The patch generated 98 new + 117 unchanged - 19 fixed = 215 total (was 136) +1 mvnsite 2m 51s the patch passed -1 whitespace 0m 0s The patch has 45 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 1s The patch 3 line(s) with tabs. +1 xml 0m 7s The patch has no ill-formed XML file. +1 shadedclient 10m 24s patch has no errors when building and testing our client artifacts. +1 findbugs 4m 36s the patch passed -1 javadoc 0m 31s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) -1 javadoc 0m 23s hadoop-tools_hadoop-aws generated 9 new + 1 unchanged - 0 fixed = 10 total (was 1)       Other Tests -1 unit 8m 49s hadoop-common in the patch failed. +1 unit 0m 53s hadoop-yarn-registry in the patch passed. +1 unit 3m 9s hadoop-mapreduce-client-core in the patch passed. +1 unit 1m 38s hadoop-aws in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 104m 34s Reason Tests Failed junit tests hadoop.http.TestHttpServer   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue HADOOP-15003 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895197/HADOOP-13786-041.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux c5e2667bdeca 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 56b88b0 maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/whitespace-tabs.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13613/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 041

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 041

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development