Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3541

Deadlock between recovery, xceiver and packet responder

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Block Recovery initiated while write in progress at Datanode side. Found a lock between recovery, xceiver and packet responder.

      1. DN_dump.rar
        7 kB
        suja s
      2. HDFS-3541.patch
        6 kB
        Vinayakumar B
      3. HDFS-3541-2.patch
        6 kB
        Vinayakumar B

        Issue Links

          Activity

          Hide
          Vinayakumar B added a comment -

          By seeing the ThreadDump attached, recoverBlock(..) call is waiting to join the writer thread in ReplicaInPipeline#stopWriter().

            public void stopWriter() throws IOException {
              if (writer != null && writer != Thread.currentThread() && writer.isAlive()) {
                writer.interrupt();
                try {
                  writer.join();
                } catch (InterruptedException e) {
                  throw new IOException("Waiting for writer thread is interrupted.");
                }
              }
            }

          FSDataSetImpl#initReplicaRecovery will call the above Method, but it have already locked the FSDataSet.

          In the current thread dump, writer thread is one of the DataXceiver threads, which are waiting on their respective PacketResponder threads.

          1. Here writer.interrupt() will succeed in interrupting the thread only in case if the it is in waiting/sleeping state. otherwise it will not actually intterrupt it. So it will wait till the thread completes its execution.
          2. writer thread is DataXceiver thread, which is waiting to join PacketResponder Thread.
          3. Packet Responders are waiting on fsdataset lock to finalize the block.

          So its a deadlock.

          Here ReplicaInPipeline#stopWriter() should ensure that thread is interrupted successfully.

          following changes should work in this case

            public void stopWriter() throws IOException {
              if (writer != null && writer != Thread.currentThread()) {
                while (writer.isAlive()) {
                  writer.interrupt();
                  try {
                    writer.wait(100);
                  } catch (InterruptedException e) {
                    throw new IOException("Waiting for writer thread is interrupted.");
                  }
                }
              }
            }
          Show
          Vinayakumar B added a comment - By seeing the ThreadDump attached, recoverBlock(..) call is waiting to join the writer thread in ReplicaInPipeline#stopWriter(). public void stopWriter() throws IOException { if (writer != null && writer != Thread .currentThread() && writer.isAlive()) { writer.interrupt(); try { writer.join(); } catch (InterruptedException e) { throw new IOException( "Waiting for writer thread is interrupted." ); } } } FSDataSetImpl#initReplicaRecovery will call the above Method, but it have already locked the FSDataSet. In the current thread dump, writer thread is one of the DataXceiver threads, which are waiting on their respective PacketResponder threads. Here writer.interrupt() will succeed in interrupting the thread only in case if the it is in waiting/sleeping state. otherwise it will not actually intterrupt it. So it will wait till the thread completes its execution. writer thread is DataXceiver thread, which is waiting to join PacketResponder Thread. Packet Responders are waiting on fsdataset lock to finalize the block. So its a deadlock. Here ReplicaInPipeline#stopWriter() should ensure that thread is interrupted successfully. following changes should work in this case public void stopWriter() throws IOException { if (writer != null && writer != Thread .currentThread()) { while (writer.isAlive()) { writer.interrupt(); try { writer.wait(100); } catch (InterruptedException e) { throw new IOException( "Waiting for writer thread is interrupted." ); } } } }
          Hide
          Robert Joseph Evans added a comment -

          I really would like to see this fixed in 0.23 as well.

          Show
          Robert Joseph Evans added a comment - I really would like to see this fixed in 0.23 as well.
          Hide
          Kihwal Lee added a comment -

          With the proposed fix, the DataXceiver thread would break out of responder.join(), but wouldn't PacketResponder keep running and do finalizeBlock() and closeBlock()? It would even try to send an ack back, which would probably fail anyway.

          Rather than letting PacketResponder asynchronously modify the state, can we make it stop?

          Show
          Kihwal Lee added a comment - With the proposed fix, the DataXceiver thread would break out of responder.join(), but wouldn't PacketResponder keep running and do finalizeBlock() and closeBlock()? It would even try to send an ack back, which would probably fail anyway. Rather than letting PacketResponder asynchronously modify the state, can we make it stop?
          Hide
          Vinayakumar B added a comment -

          Hi Lee,
          I agree your suggestion.
          But as of now there is no suggestion to stop the PacketResponder.

          Following things can happen with the proposed solution,

          1. initReplicaRecovery(..) call will interrupt the receiver thread, but it will make the replica state to RUR and release the fsdataset lock.
          2. now PacketResponder may finalize the block, i.e. replica state will be changed to FINALIZED,
          3. then updateReplicaUnderRecovery(..) call will fail because replica is not in RUR state

          I think we can restrict PacketResponder to finalize the block which is in RUR by throwing exception. In this case updateReplicaUnderRecovery(..) will not fail, and recovery will be success.

          Show
          Vinayakumar B added a comment - Hi Lee, I agree your suggestion. But as of now there is no suggestion to stop the PacketResponder. Following things can happen with the proposed solution, initReplicaRecovery(..) call will interrupt the receiver thread, but it will make the replica state to RUR and release the fsdataset lock. now PacketResponder may finalize the block, i.e. replica state will be changed to FINALIZED, then updateReplicaUnderRecovery(..) call will fail because replica is not in RUR state I think we can restrict PacketResponder to finalize the block which is in RUR by throwing exception. In this case updateReplicaUnderRecovery(..) will not fail, and recovery will be success.
          Hide
          Vinayakumar B added a comment -

          But as of now there is no way to stop the PacketResponder.

          Show
          Vinayakumar B added a comment - But as of now there is no way to stop the PacketResponder.
          Hide
          Vinayakumar B added a comment -

          I found out the Actual Problem.

          Problem resides in PacketResponder#close()

          public synchronized void close() {
                while (running && ackQueue.size() != 0 && datanode.shouldRun) {
                  try {
                    wait();
                  } catch (InterruptedException e) {
                    running = false;
                  }
                }
                if(LOG.isDebugEnabled()) {
                  LOG.debug(myString + ": closing");
                }
                running = false;
                notifyAll();
              }

          Here InterruptedException is handled but, intterrupted flag is not reset. and BlockReceiver is waiting for PacketResponder to join. But PacketResponder is BLOCKED.

          Show
          Vinayakumar B added a comment - I found out the Actual Problem. Problem resides in PacketResponder#close() public synchronized void close() { while (running && ackQueue.size() != 0 && datanode.shouldRun) { try { wait(); } catch (InterruptedException e) { running = false ; } } if (LOG.isDebugEnabled()) { LOG.debug(myString + ": closing" ); } running = false ; notifyAll(); } Here InterruptedException is handled but, intterrupted flag is not reset. and BlockReceiver is waiting for PacketResponder to join. But PacketResponder is BLOCKED.
          Hide
          Kihwal Lee added a comment -

          But as of now there is no way to stop the PacketResponder

          When (DataXciever) writer gets interrupted, it can fire an interrupt to (PacketResponder) responder. If we put a Thread.interrupted() check in finalizeBlock() and let it bail out or throw an InterruptedException, we can stop responder. Of course, PacketResponder will also need some changes to propely handle the exception. What do you think?

          Show
          Kihwal Lee added a comment - But as of now there is no way to stop the PacketResponder When (DataXciever) writer gets interrupted, it can fire an interrupt to (PacketResponder) responder. If we put a Thread.interrupted() check in finalizeBlock() and let it bail out or throw an InterruptedException, we can stop responder. Of course, PacketResponder will also need some changes to propely handle the exception. What do you think?
          Hide
          Vinayakumar B added a comment -

          Attaching the patch. Fixed as per suggestions by Lee. Please review..

          Show
          Vinayakumar B added a comment - Attaching the patch. Fixed as per suggestions by Lee. Please review..
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12533031/HDFS-3541.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestDatanodeBlockScanner

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2683//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2683//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533031/HDFS-3541.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2683//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2683//console This message is automatically generated.
          Hide
          Vinayakumar B added a comment -

          Below failures are not replated to current patch

          -1 javadoc. The javadoc tool appears to have generated 13 warning messages.
          
          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
          
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          
          
          Show
          Vinayakumar B added a comment - Below failures are not replated to current patch -1 javadoc. The javadoc tool appears to have generated 13 warning messages. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          Hide
          Aaron T. Myers added a comment -

          Patch looks pretty good to me. Just two small comments:

          1. Misspelled "interrupted": "Finalizing block from Inturrupted thread should fail"
          2. This chunk of code confuses me, since you don't use written again after the loop, and there doesn't seem to be any need to call write(...) many times:
            +      int written = 0;
            +      for (; written < 512;) {
            +        out.writeBytes(data);
            +        written += 4;
            +      }
            

          Kihwal, how does this patch look to you?

          Show
          Aaron T. Myers added a comment - Patch looks pretty good to me. Just two small comments: Misspelled "interrupted": "Finalizing block from Inturrupted thread should fail" This chunk of code confuses me, since you don't use written again after the loop, and there doesn't seem to be any need to call write(...) many times: + int written = 0; + for (; written < 512;) { + out.writeBytes(data); + written += 4; + } Kihwal, how does this patch look to you?
          Hide
          Kihwal Lee added a comment -

          The patch looks okay but I was wondering whether the test can be improved.

          The test in the current patch does not directly recreate the original race condition. Probably an artificial deadlock can be created by creating a thread which does sleep and then kills the writer inside a synchronized(datanode.data) block. While it's sleeping, another thread could try closing the DFSOutputStream. This should fail when the writer (i.e. the DataXceiver thread) is killed and streams get closed. After this we could verify the block is not finalized. Then we know the PacketResponder thread didn't finalize the block.

          Does it make sense?

          Show
          Kihwal Lee added a comment - The patch looks okay but I was wondering whether the test can be improved. The test in the current patch does not directly recreate the original race condition. Probably an artificial deadlock can be created by creating a thread which does sleep and then kills the writer inside a synchronized(datanode.data) block. While it's sleeping, another thread could try closing the DFSOutputStream . This should fail when the writer (i.e. the DataXceiver thread) is killed and streams get closed. After this we could verify the block is not finalized. Then we know the PacketResponder thread didn't finalize the block. Does it make sense?
          Hide
          Uma Maheswara Rao G added a comment -

          for the comment:

          2.This chunk of code confuses me, since you don't use written again after the loop, and there doesn't seem to be any need to call write(...) many times:

          try using util APIs already available for writing data.

          @Kihwal, good point, worth asserting block finalization.

          Show
          Uma Maheswara Rao G added a comment - for the comment: 2.This chunk of code confuses me, since you don't use written again after the loop, and there doesn't seem to be any need to call write(...) many times: try using util APIs already available for writing data. @Kihwal, good point, worth asserting block finalization.
          Hide
          Vinayakumar B added a comment -

          Attaching the patch which address above comments.
          Thanks Lee for the hint to write test to reproduce same case.

          Show
          Vinayakumar B added a comment - Attaching the patch which address above comments. Thanks Lee for the hint to write test to reproduce same case.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535219/HDFS-3541-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2742//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2742//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535219/HDFS-3541-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2742//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2742//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          The new patch looks good. I ran the new test case without the fix. It successfully deadlocked and failed. It passed with the actual fix.

          Show
          Kihwal Lee added a comment - The new patch looks good. I ran the new test case without the fix. It successfully deadlocked and failed. It passed with the actual fix.
          Hide
          Uma Maheswara Rao G added a comment -

          +1 Patch looks good to me as well. I will commit this patch in some time.

          Show
          Uma Maheswara Rao G added a comment - +1 Patch looks good to me as well. I will commit this patch in some time.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2433 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2433/)
          HDFS-3541. Deadlock between recovery, xceiver and packet responder. Contributed by Vinay.

          Submitted by: Vinay
          Reviewed by: Uma Maheswara Rao G (Revision 1358794)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2433 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2433/ ) HDFS-3541 . Deadlock between recovery, xceiver and packet responder. Contributed by Vinay. Submitted by: Vinay Reviewed by: Uma Maheswara Rao G (Revision 1358794) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2501 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2501/)
          HDFS-3541. Deadlock between recovery, xceiver and packet responder. Contributed by Vinay.

          Submitted by: Vinay
          Reviewed by: Uma Maheswara Rao G (Revision 1358794)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2501 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2501/ ) HDFS-3541 . Deadlock between recovery, xceiver and packet responder. Contributed by Vinay. Submitted by: Vinay Reviewed by: Uma Maheswara Rao G (Revision 1358794) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Hide
          Uma Maheswara Rao G added a comment -

          I have just committed this to trunk and branch-2.
          Thanks a lot Vinay for the patch.
          Also Thanks to Kihwal, Aaron for your reviews.

          Submitted by:Vivay
          Reviewed by: Uma Maheswara Rao G, Kihwal Lee, Aaron

          Show
          Uma Maheswara Rao G added a comment - I have just committed this to trunk and branch-2. Thanks a lot Vinay for the patch. Also Thanks to Kihwal, Aaron for your reviews. Submitted by:Vivay Reviewed by: Uma Maheswara Rao G, Kihwal Lee, Aaron
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Bobby,

          I really would like to see this fixed in 0.23 as well.

          Do you mind merging it to 23 branch as you showed interest to be fixed in 0.23 as well?

          Show
          Uma Maheswara Rao G added a comment - Hi Bobby, I really would like to see this fixed in 0.23 as well. Do you mind merging it to 23 branch as you showed interest to be fixed in 0.23 as well?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2451 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2451/)
          HDFS-3541. Deadlock between recovery, xceiver and packet responder. Contributed by Vinay.

          Submitted by: Vinay
          Reviewed by: Uma Maheswara Rao G (Revision 1358794)

          Result = FAILURE
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2451 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2451/ ) HDFS-3541 . Deadlock between recovery, xceiver and packet responder. Contributed by Vinay. Submitted by: Vinay Reviewed by: Uma Maheswara Rao G (Revision 1358794) Result = FAILURE umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1098 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1098/)
          HDFS-3541. Deadlock between recovery, xceiver and packet responder. Contributed by Vinay.

          Submitted by: Vinay
          Reviewed by: Uma Maheswara Rao G (Revision 1358794)

          Result = FAILURE
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1098 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1098/ ) HDFS-3541 . Deadlock between recovery, xceiver and packet responder. Contributed by Vinay. Submitted by: Vinay Reviewed by: Uma Maheswara Rao G (Revision 1358794) Result = FAILURE umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1131 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1131/)
          HDFS-3541. Deadlock between recovery, xceiver and packet responder. Contributed by Vinay.

          Submitted by: Vinay
          Reviewed by: Uma Maheswara Rao G (Revision 1358794)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1131 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1131/ ) HDFS-3541 . Deadlock between recovery, xceiver and packet responder. Contributed by Vinay. Submitted by: Vinay Reviewed by: Uma Maheswara Rao G (Revision 1358794) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Hide
          Robert Joseph Evans added a comment -

          @Uma,

          Sorry it took me so long to respond. Yes, I would be happy to look into do the porting, as the patch does not just apply. I filed HDFS-3622 to do this work on.

          Show
          Robert Joseph Evans added a comment - @Uma, Sorry it took me so long to respond. Yes, I would be happy to look into do the porting, as the patch does not just apply. I filed HDFS-3622 to do this work on.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/)
          HDFS-3541. Deadlock between recovery, xceiver and packet responder. Contributed by Vinay.

          Submitted by: Vinay
          Reviewed by: Uma Maheswara Rao G (Revision 1358794)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/ ) HDFS-3541 . Deadlock between recovery, xceiver and packet responder. Contributed by Vinay. Submitted by: Vinay Reviewed by: Uma Maheswara Rao G (Revision 1358794) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/)
          HDFS-3541. Deadlock between recovery, xceiver and packet responder. Contributed by Vinay.

          Submitted by: Vinay
          Reviewed by: Uma Maheswara Rao G (Revision 1358794)

          Result = SUCCESS
          umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/ ) HDFS-3541 . Deadlock between recovery, xceiver and packet responder. Contributed by Vinay. Submitted by: Vinay Reviewed by: Uma Maheswara Rao G (Revision 1358794) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358794 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #316 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/316/)
          HDFS-3622. Backport HDFS-3541 to branch-0.23 (bobby via daryn) (Revision 1362150)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362150
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #316 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/316/ ) HDFS-3622 . Backport HDFS-3541 to branch-0.23 (bobby via daryn) (Revision 1362150) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362150 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

            People

            • Assignee:
              Vinayakumar B
              Reporter:
              suja s
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development