Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5922

DN heartbeat thread can get stuck in tight loop

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 3.0.0, 2.4.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      We saw an issue recently on a test cluster where one of the DN threads was consuming 100% of a single CPU. Running jstack indicated that it was the DN heartbeat thread. I believe I've tracked down the cause to a bug in the accounting around the value of pendingReceivedRequests.

      More details in the first comment.

      1. HDFS-5922.01.patch
        3 kB
        Arpit Agarwal
      2. HDFS-5922.02.patch
        12 kB
        Arpit Agarwal

        Activity

        Hide
        Aaron T. Myers added a comment -

        In the heartbeat thread in BPServiceActor, we have the following:

        if (waitTime > 0 && pendingReceivedRequests == 0) {
          try {
           pendingIncrementalBRperStorage.wait(waitTime);
        

        This means that if for some reason the value of pendingReceivedRequests permanently stays positive then we will never sleep in the heartbeat thread. The question, then, is what can cause this value to stay positive.

        I believe the issue is that in BPServiceActor#addPendingReplicationBlockInfo we might not increase the size of the PerStoragePendingIncrementalBR if there is already an entry for a given block in there:

            // Make sure another entry for the same block is first removed.
            // There may only be one such entry.
            for (Map.Entry<String, PerStoragePendingIncrementalBR> entry :
                  pendingIncrementalBRperStorage.entrySet()) {
              if (entry.getValue().removeBlockInfo(bInfo)) {
                break;
              }
            }
            getIncrementalBRMapForStorage(storageUuid).putBlockInfo(bInfo);
        

        But in BPServiceActor#notifyNamenodeBlockImmediately we will always increment pendingReceivedRequests regardless of whether or not there was already an entry for the block:

          void notifyNamenodeBlockImmediately(
              ReceivedDeletedBlockInfo bInfo, String storageUuid) {
            synchronized (pendingIncrementalBRperStorage) {
              addPendingReplicationBlockInfo(bInfo, storageUuid);
              pendingReceivedRequests++;
              pendingIncrementalBRperStorage.notifyAll();
            }
          }
        

        Then, in BPServiceActor#reportReceivedDeletedBlocks, we will only subtract the number of blocks that are actually in the PerStoragePendingIncrementalBR from pendingReceivedRequests:

                  ReceivedDeletedBlockInfo[] rdbi = perStorageMap.dequeueBlockInfos();
                  pendingReceivedRequests =
                      (pendingReceivedRequests > rdbi.length ?
                          (pendingReceivedRequests - rdbi.length) : 0);
        

        This means that if we ever call BPServiceActor#notifyNamenodeBlockImmediately twice without calling BPServiceActor#reportReceivedDeletedBlocks in between, we will have pendingReceivedRequests at 2, but then only subtract 1 from it.

        Andrew Wang also pointed out offline that it is perhaps incorrect to be subtracting the number of deleted blocks from pendingReceivedRequests in BPServiceActor#reportReceivedDeletedBlocks, but the result of that is somewhat less serious, since in that case the worst case is just that we send a somewhat delayed IBR.

        Show
        Aaron T. Myers added a comment - In the heartbeat thread in BPServiceActor, we have the following: if (waitTime > 0 && pendingReceivedRequests == 0) { try { pendingIncrementalBRperStorage.wait(waitTime); This means that if for some reason the value of pendingReceivedRequests permanently stays positive then we will never sleep in the heartbeat thread. The question, then, is what can cause this value to stay positive. I believe the issue is that in BPServiceActor#addPendingReplicationBlockInfo we might not increase the size of the PerStoragePendingIncrementalBR if there is already an entry for a given block in there: // Make sure another entry for the same block is first removed. // There may only be one such entry. for (Map.Entry< String , PerStoragePendingIncrementalBR> entry : pendingIncrementalBRperStorage.entrySet()) { if (entry.getValue().removeBlockInfo(bInfo)) { break ; } } getIncrementalBRMapForStorage(storageUuid).putBlockInfo(bInfo); But in BPServiceActor#notifyNamenodeBlockImmediately we will always increment pendingReceivedRequests regardless of whether or not there was already an entry for the block: void notifyNamenodeBlockImmediately( ReceivedDeletedBlockInfo bInfo, String storageUuid) { synchronized (pendingIncrementalBRperStorage) { addPendingReplicationBlockInfo(bInfo, storageUuid); pendingReceivedRequests++; pendingIncrementalBRperStorage.notifyAll(); } } Then, in BPServiceActor#reportReceivedDeletedBlocks , we will only subtract the number of blocks that are actually in the PerStoragePendingIncrementalBR from pendingReceivedRequests : ReceivedDeletedBlockInfo[] rdbi = perStorageMap.dequeueBlockInfos(); pendingReceivedRequests = (pendingReceivedRequests > rdbi.length ? (pendingReceivedRequests - rdbi.length) : 0); This means that if we ever call BPServiceActor#notifyNamenodeBlockImmediately twice without calling BPServiceActor#reportReceivedDeletedBlocks in between, we will have pendingReceivedRequests at 2, but then only subtract 1 from it. Andrew Wang also pointed out offline that it is perhaps incorrect to be subtracting the number of deleted blocks from pendingReceivedRequests in BPServiceActor#reportReceivedDeletedBlocks , but the result of that is somewhat less serious, since in that case the worst case is just that we send a somewhat delayed IBR.
        Hide
        Arpit Agarwal added a comment -

        Hi Aaron,

        Good catch and thanks for the detailed explanation.

        I can fix it today if you haven't started. This probably needs to be in 2.3.

        Show
        Arpit Agarwal added a comment - Hi Aaron, Good catch and thanks for the detailed explanation. I can fix it today if you haven't started. This probably needs to be in 2.3.
        Hide
        Aaron T. Myers added a comment -

        Hi Arpit, yes please do take a look at fixing it. I was hoping you'd notice it since I'm less familiar with this code.

        I didn't file it as a blocker against 2.3 because the window for hitting this is really quite narrow, it's not the end of the world if a DN ends up hitting this, and I don't want to further hold up the 2.3.0 release. I personally think we should target this for 2.3.1 / 2.4.0.

        That said, if you think this is more serious than I do, then we can certainly raise the priority and target it for 2.3.0 if you want.

        Show
        Aaron T. Myers added a comment - Hi Arpit, yes please do take a look at fixing it. I was hoping you'd notice it since I'm less familiar with this code. I didn't file it as a blocker against 2.3 because the window for hitting this is really quite narrow, it's not the end of the world if a DN ends up hitting this, and I don't want to further hold up the 2.3.0 release. I personally think we should target this for 2.3.1 / 2.4.0. That said, if you think this is more serious than I do, then we can certainly raise the priority and target it for 2.3.0 if you want.
        Hide
        Arpit Agarwal added a comment -

        That sounds fine too. Thanks.

        Show
        Arpit Agarwal added a comment - That sounds fine too. Thanks.
        Hide
        Aaron T. Myers added a comment -

        Hey Arpit - any update on this one? I've started seeing it more and more on test clusters, so I think it's actually fairly likely to occur in practice if the load is high enough.

        I'll be happy to review a fix once you post one.

        Show
        Aaron T. Myers added a comment - Hey Arpit - any update on this one? I've started seeing it more and more on test clusters, so I think it's actually fairly likely to occur in practice if the load is high enough. I'll be happy to review a fix once you post one.
        Hide
        Arpit Agarwal added a comment -

        Hi Aaron, sorry about the delayed response. I was away. Here's a preliminary patch to get Jenkins results.

        The specific bug here could have been avoided by resetting the counter to zero when emptying the queues. However it seems unnecessary to maintain an exact count of the pending requests when all we care about is whether or not there are any requests. The patch replaces the counter with a boolean.

        Andrew Wang also pointed out offline that it is perhaps incorrect to be subtracting the number of deleted blocks from pendingReceivedRequests in BPServiceActor#reportReceivedDeletedBlocks, but the result of that is somewhat less serious, since in that case the worst case is just that we send a somewhat delayed IBR.

        This behavior looks odd but it was probably by design. pendingReceivedRequests was not incremented for deleted requests to avoid sending an IBR for just deleted blocks before the timeout interval has elapsed. However when we failed to send an IBR we reinserted all pending entries into the queue and set pendingReceivedRequests to be the count of all pending requests - deleted+received - presumably to avoid waiting for another timeout interval before retrying.

        Show
        Arpit Agarwal added a comment - Hi Aaron, sorry about the delayed response. I was away. Here's a preliminary patch to get Jenkins results. The specific bug here could have been avoided by resetting the counter to zero when emptying the queues. However it seems unnecessary to maintain an exact count of the pending requests when all we care about is whether or not there are any requests. The patch replaces the counter with a boolean. Andrew Wang also pointed out offline that it is perhaps incorrect to be subtracting the number of deleted blocks from pendingReceivedRequests in BPServiceActor#reportReceivedDeletedBlocks, but the result of that is somewhat less serious, since in that case the worst case is just that we send a somewhat delayed IBR. This behavior looks odd but it was probably by design. pendingReceivedRequests was not incremented for deleted requests to avoid sending an IBR for just deleted blocks before the timeout interval has elapsed. However when we failed to send an IBR we reinserted all pending entries into the queue and set pendingReceivedRequests to be the count of all pending requests - deleted+received - presumably to avoid waiting for another timeout interval before retrying.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12630522/HDFS-5922.01.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6215//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6215//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12630522/HDFS-5922.01.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6215//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6215//console This message is automatically generated.
        Hide
        Arpit Agarwal added a comment -

        Add unit test for this bug. Also added a couple of related unit tests which were missing.

        Show
        Arpit Agarwal added a comment - Add unit test for this bug. Also added a couple of related unit tests which were missing.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12630573/HDFS-5922.02.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6216//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6216//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12630573/HDFS-5922.02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6216//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6216//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12630574/HDFS-5922.02.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6217//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6217//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12630574/HDFS-5922.02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6217//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6217//console This message is automatically generated.
        Hide
        Arpit Agarwal added a comment -

        The TestCacheDirectives failure is unrelated to the patch.

        Show
        Arpit Agarwal added a comment - The TestCacheDirectives failure is unrelated to the patch.
        Hide
        Aaron T. Myers added a comment -

        +1, the latest patch looks great to me, and I agree that the test failure is unrelated.

        Thanks a lot for taking care of this, Arpit. I think this is a much simpler solution.

        Show
        Aaron T. Myers added a comment - +1, the latest patch looks great to me, and I agree that the test failure is unrelated. Thanks a lot for taking care of this, Arpit. I think this is a much simpler solution.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5219 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5219/)
        HDFS-5922. DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571542)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5219 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5219/ ) HDFS-5922 . DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571542 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
        Hide
        Arpit Agarwal added a comment -

        Thanks Aaron, and thanks again for root causing it.

        I committed it to trunk through branch-2.4.

        Show
        Arpit Agarwal added a comment - Thanks Aaron, and thanks again for root causing it. I committed it to trunk through branch-2.4.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #492 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/492/)
        HDFS-5922. DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571542)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #492 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/492/ ) HDFS-5922 . DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571542 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1684 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1684/)
        HDFS-5922. DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571542)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1684 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1684/ ) HDFS-5922 . DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571542 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1709 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1709/)
        HDFS-5922. DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571542)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1709 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1709/ ) HDFS-5922 . DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1571542 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java

          People

          • Assignee:
            Arpit Agarwal
            Reporter:
            Aaron T. Myers
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development