Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12377

Refactor TestReadStripedFileWithDecoding to avoid test timeouts

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha3
    • Fix Version/s: 3.0.0-beta1
    • Component/s: erasure-coding
    • Labels:
      None

      Description

      This test times out since the nested for loops means it runs 12 configurations inside each test method.

      Let's refactor this to use JUnit parameters instead.

      1. HDFS-12377.004.patch
        41 kB
        Andrew Wang
      2. HDFS-12377.003.patch
        42 kB
        Andrew Wang
      3. HDFS-12377.002.patch
        38 kB
        Andrew Wang
      4. HDFS-12377.001.patch
        26 kB
        Andrew Wang

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Patch attached.

        Show
        andrew.wang Andrew Wang added a comment - Patch attached.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 14m 55s trunk passed
        +1 compile 0m 54s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 findbugs 1m 42s trunk passed
        +1 javadoc 0m 41s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 54s the patch passed
        +1 compile 0m 45s the patch passed
        -1 javac 0m 45s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 408 unchanged - 3 fixed = 411 total (was 411)
        -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 19 new + 5 unchanged - 2 fixed = 24 total (was 7)
        +1 mvnsite 0m 52s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 49s the patch passed
        +1 javadoc 0m 39s the patch passed
              Other Tests
        -1 unit 94m 7s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        121m 25s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.server.namenode.TestFSImage
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.TestAclsEndToEnd
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.TestPipelines
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.TestReadStripedCorruptFileWithDecoding



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12377
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884564/HDFS-12377.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5b2d96ca9974 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3e0e203
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/20931/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20931/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20931/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20931/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20931/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 55s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 59s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 41s trunk passed       Patch Compile Tests +1 mvninstall 0m 54s the patch passed +1 compile 0m 45s the patch passed -1 javac 0m 45s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 408 unchanged - 3 fixed = 411 total (was 411) -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 19 new + 5 unchanged - 2 fixed = 24 total (was 7) +1 mvnsite 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 39s the patch passed       Other Tests -1 unit 94m 7s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 121m 25s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedCorruptFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12377 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884564/HDFS-12377.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5b2d96ca9974 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3e0e203 Default Java 1.8.0_144 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HDFS-Build/20931/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20931/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20931/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20931/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20931/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        I think we're hitting the surefire timeout (15min) vs. the junit timeout (5min) here, since I'm not seeing a stack dump. Running this test class locally takes 7.5 mins on my laptop, so we might need some further measures to avoid the timeout.

        Show
        andrew.wang Andrew Wang added a comment - I think we're hitting the surefire timeout (15min) vs. the junit timeout (5min) here, since I'm not seeing a stack dump. Running this test class locally takes 7.5 mins on my laptop, so we might need some further measures to avoid the timeout.
        Hide
        andrew.wang Andrew Wang added a comment -

        I refactored these again to use composition, and split the long-running test in half. Now runs in 4mins each on my laptop, which should slide in under the surefire timeout.

        Show
        andrew.wang Andrew Wang added a comment - I refactored these again to use composition, and split the long-running test in half. Now runs in 4mins each on my laptop, which should slide in under the surefire timeout.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 16m 8s trunk passed
        +1 compile 1m 3s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 findbugs 2m 6s trunk passed
        +1 javadoc 0m 46s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 59s the patch passed
        +1 compile 0m 56s the patch passed
        +1 javac 0m 56s hadoop-hdfs-project_hadoop-hdfs generated 0 new + 408 unchanged - 3 fixed = 408 total (was 411)
        -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 9 unchanged - 6 fixed = 16 total (was 15)
        +1 mvnsite 1m 6s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 9s the patch passed
        +1 javadoc 0m 44s the patch passed
              Other Tests
        -1 unit 99m 22s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        129m 45s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.tools.TestDFSZKFailoverController
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12377
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884817/HDFS-12377.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2d3f10ce4cc5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bac4e8c
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20948/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20948/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20948/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20948/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 8s trunk passed +1 compile 1m 3s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 4s trunk passed +1 findbugs 2m 6s trunk passed +1 javadoc 0m 46s trunk passed       Patch Compile Tests +1 mvninstall 0m 59s the patch passed +1 compile 0m 56s the patch passed +1 javac 0m 56s hadoop-hdfs-project_hadoop-hdfs generated 0 new + 408 unchanged - 3 fixed = 408 total (was 411) -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 9 unchanged - 6 fixed = 16 total (was 15) +1 mvnsite 1m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 0m 44s the patch passed       Other Tests -1 unit 99m 22s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 129m 45s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12377 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884817/HDFS-12377.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2d3f10ce4cc5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bac4e8c Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20948/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20948/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20948/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20948/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Andrew for taking care of pre-commits here! Looks pretty good to me.

        Looking at the failure, TestReadStripedFileWithDecoding#testReadWithDNFailure also needs parameterizing.
        Also, do you have insights about what's changed recently? IIRC pre-commit was fairly green at least a week ago.

        Show
        xiaochen Xiao Chen added a comment - Thanks Andrew for taking care of pre-commits here! Looks pretty good to me. Looking at the failure, TestReadStripedFileWithDecoding#testReadWithDNFailure also needs parameterizing. Also, do you have insights about what's changed recently? IIRC pre-commit was fairly green at least a week ago.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks Xiao, new patch attached parameterizing that additional test. Fixed the checkstyles and some other small improvements too along the way.

        I think the testBlockTokenExpired failures are new, and they seem to fail quite reproducibly. I haven't had time to bisect it, but agree that it seems recent.

        The EC tests in general seem very flaky, we need a push to stabilize these (particularly before GA).

        Show
        andrew.wang Andrew Wang added a comment - Thanks Xiao, new patch attached parameterizing that additional test. Fixed the checkstyles and some other small improvements too along the way. I think the testBlockTokenExpired failures are new, and they seem to fail quite reproducibly. I haven't had time to bisect it, but agree that it seems recent. The EC tests in general seem very flaky, we need a push to stabilize these (particularly before GA).
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 16m 3s trunk passed
        +1 compile 0m 57s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 findbugs 1m 58s trunk passed
        +1 javadoc 0m 42s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 58s the patch passed
        +1 compile 0m 52s the patch passed
        +1 javac 0m 52s hadoop-hdfs-project_hadoop-hdfs generated 0 new + 408 unchanged - 3 fixed = 408 total (was 411)
        -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 9 unchanged - 6 fixed = 11 total (was 15)
        +1 mvnsite 0m 58s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 9s the patch passed
        +1 javadoc 0m 40s the patch passed
              Other Tests
        -1 unit 143m 13s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 49s The patch does not generate ASF License warnings.
        173m 20s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestMaintenanceState
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.hdfs.TestDataTransferProtocol
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.TestFileAppendRestart
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.cli.TestCryptoAdminCLI
          hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.tools.TestDFSAdminWithHA
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
        Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12377
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885009/HDFS-12377.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fdf213a574e2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c5281a8
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20971/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20971/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20971/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20971/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 3s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 1s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 42s trunk passed       Patch Compile Tests +1 mvninstall 0m 58s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s hadoop-hdfs-project_hadoop-hdfs generated 0 new + 408 unchanged - 3 fixed = 408 total (was 411) -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 9 unchanged - 6 fixed = 11 total (was 15) +1 mvnsite 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 0m 40s the patch passed       Other Tests -1 unit 143m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 49s The patch does not generate ASF License warnings. 173m 20s Reason Tests Failed junit tests hadoop.hdfs.TestMaintenanceState   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDataTransferProtocol   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12377 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885009/HDFS-12377.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fdf213a574e2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c5281a8 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20971/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20971/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20971/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20971/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks for revving Andrew! +1 pending:

        • TestReadStripedFileWithDNFailure runs in the same time range as the other 2 classes (~25s on my local). I think it makes sense to raise the test case's timeout there to 5 min too, instead of the current class-level 2 min.
        • checkstyle

        Agree on improving EC tests' stability. LMK if I can help.
        It also feels to me the tests are more flaky these days not specific to EC, not sure if anything env-related changed? (or how we could find out)

        Show
        xiaochen Xiao Chen added a comment - Thanks for revving Andrew! +1 pending: TestReadStripedFileWithDNFailure runs in the same time range as the other 2 classes (~25s on my local). I think it makes sense to raise the test case's timeout there to 5 min too, instead of the current class-level 2 min. checkstyle Agree on improving EC tests' stability. LMK if I can help. It also feels to me the tests are more flaky these days not specific to EC, not sure if anything env-related changed? (or how we could find out)
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks again for reviewing Xiao, new patch to cleanup the timeout settings and also remove extra imports.

        Show
        andrew.wang Andrew Wang added a comment - Thanks again for reviewing Xiao, new patch to cleanup the timeout settings and also remove extra imports.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 45s trunk passed
        +1 compile 0m 49s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 findbugs 1m 40s trunk passed
        +1 javadoc 0m 40s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 46s the patch passed
        +1 javac 0m 46s hadoop-hdfs-project_hadoop-hdfs generated 0 new + 408 unchanged - 3 fixed = 408 total (was 411)
        +1 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 9 unchanged - 6 fixed = 9 total (was 15)
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 37s the patch passed
              Other Tests
        -1 unit 94m 28s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        120m 4s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.TestEncryptionZonesWithHA
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12377
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885455/HDFS-12377.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f4354ff8db8b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0ba8ff4
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21005/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21005/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 45s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 53s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 40s trunk passed       Patch Compile Tests +1 mvninstall 0m 49s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s hadoop-hdfs-project_hadoop-hdfs generated 0 new + 408 unchanged - 3 fixed = 408 total (was 411) +1 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 9 unchanged - 6 fixed = 9 total (was 15) +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 94m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 120m 4s Reason Tests Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12377 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885455/HDFS-12377.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f4354ff8db8b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0ba8ff4 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21005/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21005/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks again for the review Xiao, committed to trunk and branch-3.0.

        Show
        andrew.wang Andrew Wang added a comment - Thanks again for the review Xiao, committed to trunk and branch-3.0.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12811/)
        HDFS-12377. Refactor TestReadStripedFileWithDecoding to avoid test (wang: rev d4035d42f02507bc89adce3f0450c36b58b201c1)

        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecodingCorruptData.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDNFailure.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecodingDeletedData.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/ReadStripedFileWithDecodingHelper.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12811/ ) HDFS-12377 . Refactor TestReadStripedFileWithDecoding to avoid test (wang: rev d4035d42f02507bc89adce3f0450c36b58b201c1) (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecodingCorruptData.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDNFailure.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecodingDeletedData.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/ReadStripedFileWithDecodingHelper.java

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development