Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12349

Improve log message when it could not alloc enough blocks for EC

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha3
    • Fix Version/s: 3.0.0-beta1
    • Component/s: erasure-coding
    • Labels:
      None

      Description

      When an EC output stream could not alloc enough blocks for parity blocks, it sets the warning.

      if (blocks[i] == null) {
              LOG.warn("Failed to get block location for parity block, index=" + i);
      

      We should clarify the cause of this warning message.

      1. HDFS-12349.00.patch
        11 kB
        Lei (Eddy) Xu
      2. HDFS-12349.01.patch
        12 kB
        Lei (Eddy) Xu
      3. HDFS-12349.02.patch
        12 kB
        Lei (Eddy) Xu
      4. HDFS-12349.03.patch
        14 kB
        Lei (Eddy) Xu
      5. HDFS-12349.04.patch
        15 kB
        Lei (Eddy) Xu

        Activity

        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Attach the patch to fix cases for both no enough data block and no enough parity block cases.
        It also move the logic of checking data blocks to BlockManager, similar to what NN checks minReplication.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Attach the patch to fix cases for both no enough data block and no enough parity block cases. It also move the logic of checking data blocks to BlockManager , similar to what NN checks minReplication .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 26s Maven dependency ordering for branch
        +1 mvninstall 15m 26s trunk passed
        +1 compile 1m 40s trunk passed
        +1 checkstyle 0m 48s trunk passed
        +1 mvnsite 1m 38s trunk passed
        +1 findbugs 3m 22s trunk passed
        +1 javadoc 1m 6s trunk passed
              Patch Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 28s the patch passed
        +1 compile 1m 35s the patch passed
        +1 javac 1m 35s the patch passed
        -0 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 2 new + 162 unchanged - 0 fixed = 164 total (was 162)
        +1 mvnsite 1m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 34s the patch passed
        +1 javadoc 1m 3s the patch passed
              Other Tests
        +1 unit 1m 20s hadoop-hdfs-client in the patch passed.
        -1 unit 98m 47s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        136m 53s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12349
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883629/HDFS-12349.00.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b3357266f1da 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4b2c442
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20866/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20866/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20866/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20866/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 15m 26s trunk passed +1 compile 1m 40s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 1m 38s trunk passed +1 findbugs 3m 22s trunk passed +1 javadoc 1m 6s trunk passed       Patch Compile Tests 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 1m 35s the patch passed +1 javac 1m 35s the patch passed -0 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 2 new + 162 unchanged - 0 fixed = 164 total (was 162) +1 mvnsite 1m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 34s the patch passed +1 javadoc 1m 3s the patch passed       Other Tests +1 unit 1m 20s hadoop-hdfs-client in the patch passed. -1 unit 98m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 136m 53s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12349 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883629/HDFS-12349.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b3357266f1da 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4b2c442 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20866/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20866/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20866/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20866/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Many of the failures are due to error message change. Working on the fix.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Many of the failures are due to error message change. Working on the fix.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Fixed testAddBlockWhenNoSufficientDataBlockNumOfNodes failures

        Show
        eddyxu Lei (Eddy) Xu added a comment - Fixed testAddBlockWhenNoSufficientDataBlockNumOfNodes failures
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 13s Maven dependency ordering for branch
        +1 mvninstall 15m 1s trunk passed
        +1 compile 1m 33s trunk passed
        +1 checkstyle 0m 45s trunk passed
        +1 mvnsite 1m 35s trunk passed
        +1 findbugs 3m 14s trunk passed
        +1 javadoc 1m 6s trunk passed
              Patch Compile Tests
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 24s the patch passed
        +1 compile 1m 39s the patch passed
        +1 javac 1m 39s the patch passed
        -0 checkstyle 0m 46s hadoop-hdfs-project: The patch generated 2 new + 149 unchanged - 0 fixed = 151 total (was 149)
        +1 mvnsite 1m 35s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 41s the patch passed
        +1 javadoc 1m 9s the patch passed
              Other Tests
        +1 unit 1m 21s hadoop-hdfs-client in the patch passed.
        -1 unit 87m 15s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        124m 43s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12349
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884293/HDFS-12349.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6d1c681b2ac6 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1f6dc4e
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20910/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20910/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20910/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20910/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 15m 1s trunk passed +1 compile 1m 33s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 35s trunk passed +1 findbugs 3m 14s trunk passed +1 javadoc 1m 6s trunk passed       Patch Compile Tests 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 24s the patch passed +1 compile 1m 39s the patch passed +1 javac 1m 39s the patch passed -0 checkstyle 0m 46s hadoop-hdfs-project: The patch generated 2 new + 149 unchanged - 0 fixed = 151 total (was 149) +1 mvnsite 1m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 41s the patch passed +1 javadoc 1m 9s the patch passed       Other Tests +1 unit 1m 21s hadoop-hdfs-client in the patch passed. -1 unit 87m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 124m 43s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12349 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884293/HDFS-12349.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6d1c681b2ac6 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1f6dc4e Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20910/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20910/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20910/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20910/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for working on this Eddy, nice supportability improvement. I ran the changed test in TestDFSStripedOSWF and saw this modified log message:

        java.io.IOException: File /TestDFSStripedOutputStreamWithFailure/ecfile could only be replicated to 5 nodes instead of required data units for RS-6-3-1024k (=6). There are 5 datanode(s) running and no node(s) are excluded in this operation.
        

        One comment on this is that "replicated" is inaccurate for EC files, maybe say "written" to be more generic? I'd also put the 6 closer to the 5. Ex: "could only be written to 5 of the 6 required locations for RS-6-3-1024k."

        Could you provide a sample of the other modified log message? One typo: "no enough" -> "not enough"

        Show
        andrew.wang Andrew Wang added a comment - Thanks for working on this Eddy, nice supportability improvement. I ran the changed test in TestDFSStripedOSWF and saw this modified log message: java.io.IOException: File /TestDFSStripedOutputStreamWithFailure/ecfile could only be replicated to 5 nodes instead of required data units for RS-6-3-1024k (=6). There are 5 datanode(s) running and no node(s) are excluded in this operation. One comment on this is that "replicated" is inaccurate for EC files, maybe say "written" to be more generic? I'd also put the 6 closer to the 5. Ex: "could only be written to 5 of the 6 required locations for RS-6-3-1024k." Could you provide a sample of the other modified log message? One typo: "no enough" -> "not enough"
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Hi, Andrew Wang

        Thanks for the review, addressed your comments accordingly.

        The following are a few message changes:

        • "Failed to get block location for parity block, index=3" to
          "Cannot allocate parity block(index=3, policy=RS-6-3-1024k), Not enough datanodes? Exclude nodes=None)"
        • "Failed to get following block, i=3" to "Failed to allocate parity block, index=3"
        • For a 3x replicated file: "File /foo could only be written to 1 of the 2 minReplication. There are 5 datanode(s) running and no node(s) are excluded in this operation."
        • For an EC file: "File /bar could only be written to 5 of the 6 required nodes for RS-6-3-1024k. There are 5 datanode(s) running and no node(s) are excluded in this operation."
        Show
        eddyxu Lei (Eddy) Xu added a comment - Hi, Andrew Wang Thanks for the review, addressed your comments accordingly. The following are a few message changes: "Failed to get block location for parity block, index=3" to "Cannot allocate parity block(index=3, policy=RS-6-3-1024k), Not enough datanodes? Exclude nodes=None)" "Failed to get following block, i=3" to "Failed to allocate parity block, index=3" For a 3x replicated file: "File /foo could only be written to 1 of the 2 minReplication. There are 5 datanode(s) running and no node(s) are excluded in this operation." For an EC file: "File /bar could only be written to 5 of the 6 required nodes for RS-6-3-1024k. There are 5 datanode(s) running and no node(s) are excluded in this operation."
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for the rev Eddy, LGTM +1 pending this little nit:

        • "Exclude nodes" -> "Excluded nodes"
        Show
        andrew.wang Andrew Wang added a comment - Thanks for the rev Eddy, LGTM +1 pending this little nit: "Exclude nodes" -> "Excluded nodes"
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 21s Maven dependency ordering for branch
        +1 mvninstall 16m 37s trunk passed
        +1 compile 2m 0s trunk passed
        +1 checkstyle 0m 51s trunk passed
        +1 mvnsite 2m 0s trunk passed
        +1 findbugs 3m 42s trunk passed
        +1 javadoc 1m 9s trunk passed
              Patch Compile Tests
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 52s the patch passed
        +1 compile 2m 8s the patch passed
        +1 javac 2m 8s the patch passed
        -0 checkstyle 0m 49s hadoop-hdfs-project: The patch generated 2 new + 149 unchanged - 0 fixed = 151 total (was 149)
        +1 mvnsite 1m 52s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 2m 23s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 1m 9s the patch passed
              Other Tests
        +1 unit 1m 31s hadoop-hdfs-client in the patch passed.
        -1 unit 128m 54s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        172m 4s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs
          Format-string method String.format(String, Object[]) called with format string "File %s could only be written to %d of the %d %s. There are %d datanode(s) running and %s node(s) are excluded in this operation." wants 6 arguments but is given 7 in org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(String, int, Node, Set, long, List, byte, BlockType, ErasureCodingPolicy, EnumSet) At BlockManager.java:with format string "File %s could only be written to %d of the %d %s. There are %d datanode(s) running and %s node(s) are excluded in this operation." wants 6 arguments but is given 7 in org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(String, int, Node, Set, long, List, byte, BlockType, ErasureCodingPolicy, EnumSet) At BlockManager.java:[line 2076]
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.server.namenode.TestReencryptionWithKMS
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
          hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.hdfs.TestDataTransferProtocol
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.tools.TestHdfsConfigFields
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.TestBlocksScheduledCounter
          hadoop.hdfs.TestEncryptedTransfer
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          hadoop.hdfs.TestFileStatus
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12349
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885922/HDFS-12349.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 77339efd1024 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f155ab7
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21039/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/21039/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21039/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21039/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21039/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 16m 37s trunk passed +1 compile 2m 0s trunk passed +1 checkstyle 0m 51s trunk passed +1 mvnsite 2m 0s trunk passed +1 findbugs 3m 42s trunk passed +1 javadoc 1m 9s trunk passed       Patch Compile Tests 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 2m 8s the patch passed +1 javac 2m 8s the patch passed -0 checkstyle 0m 49s hadoop-hdfs-project: The patch generated 2 new + 149 unchanged - 0 fixed = 151 total (was 149) +1 mvnsite 1m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 2m 23s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 9s the patch passed       Other Tests +1 unit 1m 31s hadoop-hdfs-client in the patch passed. -1 unit 128m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 172m 4s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Format-string method String.format(String, Object[]) called with format string "File %s could only be written to %d of the %d %s. There are %d datanode(s) running and %s node(s) are excluded in this operation." wants 6 arguments but is given 7 in org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(String, int, Node, Set, long, List, byte, BlockType, ErasureCodingPolicy, EnumSet) At BlockManager.java:with format string "File %s could only be written to %d of the %d %s. There are %d datanode(s) running and %s node(s) are excluded in this operation." wants 6 arguments but is given 7 in org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(String, int, Node, Set, long, List, byte, BlockType, ErasureCodingPolicy, EnumSet) At BlockManager.java: [line 2076] Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.TestReencryptionWithKMS   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDataTransferProtocol   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestReconstructStripedFile   hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestBlocksScheduledCounter   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.TestFileStatus   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestClientProtocolForPipelineRecovery Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12349 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885922/HDFS-12349.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 77339efd1024 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f155ab7 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21039/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/21039/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/21039/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21039/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21039/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Thanks for the reviews, Andrew Wang

        Verified that all failed tests are not related. Committed to trunk.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Thanks for the reviews, Andrew Wang Verified that all failed tests are not related. Committed to trunk.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12817 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12817/)
        HDFS-12349. Improve log message when it could not alloc enough blocks (lei: rev 3e6d0ca2b2f79bfa87faa7bbd46d814a48334fbd)

        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12817 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12817/ ) HDFS-12349 . Improve log message when it could not alloc enough blocks (lei: rev 3e6d0ca2b2f79bfa87faa7bbd46d814a48334fbd) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        Hide
        chris.douglas Chris Douglas added a comment -

        Verified that all failed tests are not related

        The failed tests are related. For example, TestSaslDataTransfer

        Expected output to match /could only be replicated to 0 nodes/ but got:
        File /file1 could only be written to 0 of the 1 minReplication. There are 1 datanode(s) running and 3 node(s) are excluded in this operation.
        

        Please also fix the findbugs warning added in this patch.

        Show
        chris.douglas Chris Douglas added a comment - Verified that all failed tests are not related The failed tests are related. For example, TestSaslDataTransfer Expected output to match /could only be replicated to 0 nodes/ but got: File /file1 could only be written to 0 of the 1 minReplication. There are 1 datanode(s) running and 3 node(s) are excluded in this operation. Please also fix the findbugs warning added in this patch.
        Hide
        chris.douglas Chris Douglas added a comment -

        There's too much noise in the HDFS tests right now. Filed HDFS-12417 for the most common offender.

        Lei (Eddy) Xu, this is a small enough patch, would it make sense to revert it? The findbugs warning and failing tests are showing up in all runs of test-patch.

        Show
        chris.douglas Chris Douglas added a comment - There's too much noise in the HDFS tests right now. Filed HDFS-12417 for the most common offender. Lei (Eddy) Xu , this is a small enough patch, would it make sense to revert it? The findbugs warning and failing tests are showing up in all runs of test-patch.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Sure, Chris Douglas.

        I will revert it and make a new patch. Thanks for taking care of HDFS failures in HDFS-12417.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Sure, Chris Douglas . I will revert it and make a new patch. Thanks for taking care of HDFS failures in HDFS-12417 .
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12833 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12833/)
        Revert "HDFS-12349. Improve log message when it could not alloc enough (lei: rev de9994bd893af70fffdd68af6252fc45020e0e69)

        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12833 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12833/ ) Revert " HDFS-12349 . Improve log message when it could not alloc enough (lei: rev de9994bd893af70fffdd68af6252fc45020e0e69) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Rework to fix test failures and checkstyle warnings.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Rework to fix test failures and checkstyle warnings.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 12s Maven dependency ordering for branch
        +1 mvninstall 14m 57s trunk passed
        +1 compile 1m 39s trunk passed
        +1 checkstyle 0m 51s trunk passed
        +1 mvnsite 1m 33s trunk passed
        +1 findbugs 3m 4s trunk passed
        +1 javadoc 1m 2s trunk passed
              Patch Compile Tests
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 22s the patch passed
        +1 compile 1m 24s the patch passed
        +1 javac 1m 24s the patch passed
        +1 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 0 new + 203 unchanged - 6 fixed = 203 total (was 209)
        +1 mvnsite 1m 23s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 16s the patch passed
        +1 javadoc 0m 57s the patch passed
              Other Tests
        +1 unit 1m 12s hadoop-hdfs-client in the patch passed.
        -1 unit 100m 48s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 28s The patch does not generate ASF License warnings.
        136m 42s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestEncryptedTransfer
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.server.namenode.TestReencryptionWithKMS
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          hadoop.hdfs.TestEncryptionZonesWithHA
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12349
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886502/HDFS-12349.03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c02e6809607b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 088a18b
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21076/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21076/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21076/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 14m 57s trunk passed +1 compile 1m 39s trunk passed +1 checkstyle 0m 51s trunk passed +1 mvnsite 1m 33s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 2s trunk passed       Patch Compile Tests 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 1m 24s the patch passed +1 javac 1m 24s the patch passed +1 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 0 new + 203 unchanged - 6 fixed = 203 total (was 209) +1 mvnsite 1m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 0m 57s the patch passed       Other Tests +1 unit 1m 12s hadoop-hdfs-client in the patch passed. -1 unit 100m 48s hadoop-hdfs in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 136m 42s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.namenode.TestReencryptionWithKMS   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12349 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886502/HDFS-12349.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c02e6809607b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 088a18b Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21076/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21076/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21076/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 7s Maven dependency ordering for branch
        +1 mvninstall 14m 53s trunk passed
        +1 compile 1m 29s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 1m 31s trunk passed
        +1 findbugs 3m 6s trunk passed
        +1 javadoc 1m 5s trunk passed
              Patch Compile Tests
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 28s the patch passed
        +1 compile 1m 30s the patch passed
        +1 javac 1m 30s the patch passed
        +1 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 0 new + 203 unchanged - 6 fixed = 203 total (was 209)
        +1 mvnsite 1m 27s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 24s the patch passed
        +1 javadoc 1m 1s the patch passed
              Other Tests
        +1 unit 1m 14s hadoop-hdfs-client in the patch passed.
        -1 unit 94m 5s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        130m 18s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.namenode.TestReencryptionWithKMS
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.TestFileAppendRestart
          hadoop.hdfs.TestReplication
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12349
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886502/HDFS-12349.03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 13a32e718919 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7ee02d1
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21141/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21141/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21141/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 14m 53s trunk passed +1 compile 1m 29s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 31s trunk passed +1 findbugs 3m 6s trunk passed +1 javadoc 1m 5s trunk passed       Patch Compile Tests 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 1m 30s the patch passed +1 javac 1m 30s the patch passed +1 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 0 new + 203 unchanged - 6 fixed = 203 total (was 209) +1 mvnsite 1m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 1m 1s the patch passed       Other Tests +1 unit 1m 14s hadoop-hdfs-client in the patch passed. -1 unit 94m 5s hadoop-hdfs in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 130m 18s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.namenode.TestReencryptionWithKMS   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.TestReplication Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12349 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886502/HDFS-12349.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 13a32e718919 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ee02d1 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21141/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21141/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21141/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Fix TestBlockStatsMXBean

        Show
        eddyxu Lei (Eddy) Xu added a comment - Fix TestBlockStatsMXBean
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 12s Maven dependency ordering for branch
        +1 mvninstall 15m 23s trunk passed
        +1 compile 1m 35s trunk passed
        +1 checkstyle 0m 45s trunk passed
        +1 mvnsite 1m 31s trunk passed
        +1 findbugs 3m 16s trunk passed
        +1 javadoc 1m 12s trunk passed
              Patch Compile Tests
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 36s the patch passed
        +1 compile 1m 27s the patch passed
        +1 javac 1m 27s the patch passed
        +1 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 0 new + 214 unchanged - 6 fixed = 214 total (was 220)
        +1 mvnsite 1m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 21s the patch passed
        +1 javadoc 0m 57s the patch passed
              Other Tests
        +1 unit 1m 13s hadoop-hdfs-client in the patch passed.
        -1 unit 105m 24s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        142m 46s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSInotifyEventInputStream
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12349
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887205/HDFS-12349.04.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux eb3d3db6c544 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 390c2b5
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21146/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21146/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21146/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 15m 23s trunk passed +1 compile 1m 35s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 31s trunk passed +1 findbugs 3m 16s trunk passed +1 javadoc 1m 12s trunk passed       Patch Compile Tests 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 36s the patch passed +1 compile 1m 27s the patch passed +1 javac 1m 27s the patch passed +1 checkstyle 0m 42s hadoop-hdfs-project: The patch generated 0 new + 214 unchanged - 6 fixed = 214 total (was 220) +1 mvnsite 1m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 21s the patch passed +1 javadoc 0m 57s the patch passed       Other Tests +1 unit 1m 13s hadoop-hdfs-client in the patch passed. -1 unit 105m 24s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 142m 46s Reason Tests Failed junit tests hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12349 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12887205/HDFS-12349.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux eb3d3db6c544 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 390c2b5 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21146/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21146/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21146/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        TestNameNodeMetrics is not related, and tracked in HDFS-12456. TestLeaseRecoveryStriped and TestReconstructStripedFile are also tracked in other JIRAs.

        The rest pass locally.

        Chris Douglas, Andrew Wang could you give another review?

        Show
        eddyxu Lei (Eddy) Xu added a comment - TestNameNodeMetrics is not related, and tracked in HDFS-12456 . TestLeaseRecoveryStriped and TestReconstructStripedFile are also tracked in other JIRAs. The rest pass locally. Chris Douglas , Andrew Wang could you give another review?
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM. Could you also file a JIRA to add a JUnit timeout to TestBlockStatsMXBean? Right now it's failing due to the surefire timeout which makes it much harder to debug.

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM. Could you also file a JIRA to add a JUnit timeout to TestBlockStatsMXBean? Right now it's failing due to the surefire timeout which makes it much harder to debug.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Thanks, Andrew Wang Committed to trunk and branch-3

        Show
        eddyxu Lei (Eddy) Xu added a comment - Thanks, Andrew Wang Committed to trunk and branch-3
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12887 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12887/)
        HDFS-12349. Improve log message when it could not alloc enough blocks (lei: rev fbe06b58805aac4861fb27dfa273914b69e8bdc6)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/TestSaslDataTransfer.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12887 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12887/ ) HDFS-12349 . Improve log message when it could not alloc enough blocks (lei: rev fbe06b58805aac4861fb27dfa273914b69e8bdc6) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/TestSaslDataTransfer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

          People

          • Assignee:
            eddyxu Lei (Eddy) Xu
            Reporter:
            eddyxu Lei (Eddy) Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development