Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8245

Standby namenode doesn't process DELETED_BLOCK if the add block request is in edit log.

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The following series of events happened on Standby namenode :
      2015-04-09 07:47:21,735 [Edit log tailer] INFO ha.EditLogTailer: Triggering log roll on remote NameNode Active Namenode (ANN)
      2015-04-09 07:58:01,858 [Edit log tailer] INFO ha.EditLogTailer: Triggering log roll on remote NameNode ANN

      The following series of events happened on Active Namenode:,
      2015-04-09 07:47:21,747 [IPC Server handler 99 on 8020] INFO namenode.FSNamesystem: Roll Edit Log from Standby NN (SNN)
      2015-04-09 07:58:01,868 [IPC Server handler 18 on 8020] INFO namenode.FSNamesystem: Roll Edit Log from SNN

      The following series of events happened on datanode ( datanodeA ):
      2015-04-09 07:52:15,817 [DataXceiver for client DFSClient_attempt_1428022041757_102831_r_000107_0_1139131345_1 at /xxxx:51078 [Receiving block BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867]] INFO datanode.DataNode: Receiving BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867 src: /client:51078 dest: /datanodeA:1004

      2015-04-09 07:52:15,969 [PacketResponder: BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO DataNode.clienttrace: src: /client:51078, dest: /datanodeA:1004, bytes: 20, op: HDFS_WRITE, cliID: DFSClient_attempt_1428022041757_102831_r_000107_0_1139131345_1, offset: 0, srvID: 356a8a98-826f-446d-8f4c-ce288c1f0a75, blockid: BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867, duration: 148948385
      2015-04-09 07:52:15,969 [PacketResponder: BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO datanode.DataNode: PacketResponder: BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
      2015-04-09 07:52:25,970 [DataXceiver for client /<datanodeB> :52827 [Copying block BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867]] INFO datanode.DataNode: Copied BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867 to <datanodeB>:52827
      2015-04-09 07:52:28,187 [DataNode: heartbeating to ANN:8020] INFO impl.FsDatasetAsyncDiskService: Scheduling blk_1570321882_1102029183867 file <path>/blk_1570321882 for deletion
      2015-04-09 07:52:28,188 [Async disk worker #1482 for volume ] INFO impl.FsDatasetAsyncDiskService: Deleted BP-595383232-xxxx-1360869396230 blk_1570321882_1102029183867 file <path>/blk_1570321882

      Then we failover for upgrade and then the standby became active.

      When we did ls command on this file, we got the following exception:

      15/04/09 22:07:39 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
      java.io.IOException: Got error for OP_READ_BLOCK, self=/client:32947, remote=datanodeA:1004, for file <filename>, for pool BP-595383232-xxxx-1360869396230 block 1570321882_1102029183867
      at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:445)
      at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:410)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:815)
      at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:693)
      at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:351)
      at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:576)
      at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:800)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:847)
      at java.io.DataInputStream.read(DataInputStream.java:100)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:78)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
      at org.apache.hadoop.fs.shell.CopyCommands$Merge.processArguments(CopyCommands.java:97)
      at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
      at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

      Eventually this request was served by datanodeB on the second attempt,
      15/04/09 22:07:39 INFO hdfs.DFSClient: Successfully connected to datanodeB:1004 for BP-595383232-xxxx-1360869396230:blk_1570321882_1102029183867

      On further analyzing this problem, I found that when datanodeA notified the namenode after deleting that block, the then ANN didn't queue delete request to PendingDNMessages.
      So when the then SNN read the edits, it didn't knew that the block from datanodeA was deleted.

      1. HDFS-8245.patch
        9 kB
        Rushabh S Shah
      2. HDFS-8245-1.patch
        9 kB
        Rushabh S Shah

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Sangjin Lee backported this to 2.6.1, after fixing minor issues.

        I just pushed the commit to 2.6.1 after running compilation and TestBlockReplacement, TestDNFencing which changed in the patch.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Sangjin Lee backported this to 2.6.1, after fixing minor issues. I just pushed the commit to 2.6.1 after running compilation and TestBlockReplacement, TestDNFencing which changed in the patch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/)
        HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/ ) HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/)
        HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/ ) HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/)
        HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/ ) HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/)
        HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/ ) HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/)
        HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/ ) HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/)
        HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/ ) HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7779/)
        HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7779/ ) HDFS-8245 . Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. (kihwal: rev 2d4ae3d18bc530fa9f81ee616db8af3395705fb9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Thanks Kihwal Lee for reviewing and committing.

        Show
        shahrs87 Rushabh S Shah added a comment - Thanks Kihwal Lee for reviewing and committing.
        Hide
        kihwal Kihwal Lee added a comment -

        Committed to trunk, branch-2 and branch-2.7. Thanks for reporting, analyzing and fixing the issue, Rushabh.

        Show
        kihwal Kihwal Lee added a comment - Committed to trunk, branch-2 and branch-2.7. Thanks for reporting, analyzing and fixing the issue, Rushabh.
        Hide
        kihwal Kihwal Lee added a comment -

        +1 the patch looks good.

        Show
        kihwal Kihwal Lee added a comment - +1 the patch looks good.
        Hide
        kihwal Kihwal Lee added a comment -

        TestTraceAdmin is being fixed in HDFS-8358.
        The check style warning is about one indentation error. Rather than resubmitting a patch and wasting the build resource, we will fix it during commit.

        Show
        kihwal Kihwal Lee added a comment - TestTraceAdmin is being fixed in HDFS-8358 . The check style warning is about one indentation error. Rather than resubmitting a patch and wasting the build resource, we will fix it during commit.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 36s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 31s There were no new javac warning messages.
        +1 javadoc 9m 32s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 11s The applied patch generated 2 new checkstyle issues (total was 209, now 210).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 12s Pre-build of native portion
        -1 hdfs tests 165m 6s Tests failed in hadoop-hdfs.
            207m 46s  



        Reason Tests
        Failed unit tests hadoop.tracing.TestTraceAdmin



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731477/HDFS-8245-1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 3cefc02
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10877/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10877/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10877/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10877/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 11s The applied patch generated 2 new checkstyle issues (total was 209, now 210). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 165m 6s Tests failed in hadoop-hdfs.     207m 46s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731477/HDFS-8245-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3cefc02 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10877/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10877/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10877/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10877/console This message was automatically generated.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Thanks Kihwal Lee for the review.
        All the tests pass on my local box and they fail due to timeouts.
        Please review the new patch.
        It fixes the checkstyle and whitespace warnings otherwise the patch is the same as before.

        Show
        shahrs87 Rushabh S Shah added a comment - Thanks Kihwal Lee for the review. All the tests pass on my local box and they fail due to timeouts. Please review the new patch. It fixes the checkstyle and whitespace warnings otherwise the patch is the same as before.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Attached a new patch fixing checkstyle and whitespace warnings.
        All the test failures seems unrelated to my patch.
        Ran them all on my local box and all of them pass.

        Show
        shahrs87 Rushabh S Shah added a comment - Attached a new patch fixing checkstyle and whitespace warnings. All the test failures seems unrelated to my patch. Ran them all on my local box and all of them pass.
        Hide
        kihwal Kihwal Lee added a comment -

        Can you retest or justify unit test failures? The change looks good. The findbugs warning seems unrelated.

        Show
        kihwal Kihwal Lee added a comment - Can you retest or justify unit test failures? The change looks good. The findbugs warning seems unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 32s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 23s There were no new javac warning messages.
        +1 javadoc 9m 33s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 22s The applied patch generated 3 new checkstyle issues (total was 206, now 208).
        -1 whitespace 0m 1s The patch has 22 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        -1 findbugs 3m 5s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        -1 hdfs tests 226m 48s Tests failed in hadoop-hdfs.
            268m 30s  



        Reason Tests
        FindBugs module:hadoop-hdfs
          Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived from an Exception, even though it is named as such At DataStreamer.java:from an Exception, even though it is named as such At DataStreamer.java:[lines 177-201]
        Failed unit tests hadoop.hdfs.server.namenode.TestDeleteRace
          hadoop.hdfs.web.TestWebHDFS
          hadoop.hdfs.TestClose
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.TestQuota
          hadoop.hdfs.TestMultiThreadedHflush
          hadoop.hdfs.TestDFSOutputStream
          hadoop.hdfs.server.namenode.TestSaveNamespace
          hadoop.hdfs.server.datanode.TestBlockRecovery
          hadoop.cli.TestHDFSCLI
          hadoop.hdfs.TestFileLengthOnClusterRestart
          hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
        Timed out tests org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.TestDataTransferProtocol



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729637/HDFS-8245.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 87e9978
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10498/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10498/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/10498/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10498/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10498/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10498/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 23s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 3 new checkstyle issues (total was 206, now 208). -1 whitespace 0m 1s The patch has 22 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. -1 findbugs 3m 5s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 226m 48s Tests failed in hadoop-hdfs.     268m 30s   Reason Tests FindBugs module:hadoop-hdfs   Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived from an Exception, even though it is named as such At DataStreamer.java:from an Exception, even though it is named as such At DataStreamer.java: [lines 177-201] Failed unit tests hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.TestClose   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.TestQuota   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.TestDFSOutputStream   hadoop.hdfs.server.namenode.TestSaveNamespace   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.cli.TestHDFSCLI   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation Timed out tests org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache   org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery   org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   org.apache.hadoop.hdfs.TestDataTransferProtocol Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729637/HDFS-8245.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 87e9978 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10498/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10498/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/10498/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10498/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10498/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10498/console This message was automatically generated.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        I have just added a check in BlockManager#removeStoredBlock to add to pendingDNMessages if the namenode is standby.

        There is one test case failing TestDNFencing#testDnFencing by the change.
        The test case was checking for postponedMisreplicatedBlocksCount after the failover.
        Since the change adds the deleted block to the pending messages queue and during the trnasition to active, it dequeues the deleted block request.
        So the postponedMisreplicatedBlocksCount will be zero after the failOver.

        Show
        shahrs87 Rushabh S Shah added a comment - I have just added a check in BlockManager#removeStoredBlock to add to pendingDNMessages if the namenode is standby. There is one test case failing TestDNFencing#testDnFencing by the change. The test case was checking for postponedMisreplicatedBlocksCount after the failover. Since the change adds the deleted block to the pending messages queue and during the trnasition to active, it dequeues the deleted block request. So the postponedMisreplicatedBlocksCount will be zero after the failOver.

          People

          • Assignee:
            shahrs87 Rushabh S Shah
            Reporter:
            shahrs87 Rushabh S Shah
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development