Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7999

FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time

    Details

    • Target Version/s:

      Description

      I'm using 2.6.0 and noticed that sometime DN's heartbeat were delayed for very long time, say more than 100 seconds. I get the jstack twice and looks like they are all blocked (at getStorageReport) by dataset lock, and which is held by a thread that is calling createTemporary, which again is blocked to wait earlier incarnation writer to exit.

      The heartbeat thread stack:
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:152)

      • waiting to lock <0x00000007b01428c0> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:144)
      • locked <0x00000007b0140ed0> (a java.lang.Object)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:575)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:680)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:850)
        at java.lang.Thread.run(Thread.java:662)

      The DataXceiver thread holds the dataset lock:
      "DataXceiver for client at XXXXX" daemon prio=10 tid=0x00007f14041e6480 nid=0x52bc in Object.wait() [0x00007f11d78f7000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Thread.join(Thread.java:1194)
      locked <0x00000007a33b85d8> (a org.apache.hadoop.util.Daemon)
      at org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:183)
      at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:1231)
      locked <0x00000007b01428c0> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
      at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:114)
      at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:179)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
      at java.lang.Thread.run(Thread.java:662)

      1. HDFS-7999-001.patch
        4 kB
        zhouyingchao
      2. HDFS-7999-002.patch
        4 kB
        zhouyingchao
      3. HDFS-7999-003.patch
        4 kB
        zhouyingchao
      4. HDFS-7999-branch-2.6.1.txt
        5 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          sinago zhouyingchao added a comment -

          The fix is to call stopWriter w/o the FsDatasetImpl lock. However, without lock, another thread may slip in and inject another ReplicaInfo to the map when we stop the writter. To resolve the issue, we will try to invalidate stale replica in a loop. As the last resort, if we hang in the thread too long, we will bail out the loop with an IOException.

          Show
          sinago zhouyingchao added a comment - The fix is to call stopWriter w/o the FsDatasetImpl lock. However, without lock, another thread may slip in and inject another ReplicaInfo to the map when we stop the writter. To resolve the issue, we will try to invalidate stale replica in a loop. As the last resort, if we hang in the thread too long, we will bail out the loop with an IOException.
          Hide
          sinago zhouyingchao added a comment -

          Test with "-Dtest=FsDatasetTestUtil,LazyPersistTestCase,TestDatanodeRestart,TestFsDatasetImpl,TestFsVolumeList,TestInterDatanodeProtocol,TestLazyPersistFiles,TestRbwSpaceReservation,TestReplicaMap,TestScrLazyPersistFiles,TestWriteToReplica"

          Show
          sinago zhouyingchao added a comment - Test with "-Dtest=FsDatasetTestUtil,LazyPersistTestCase,TestDatanodeRestart,TestFsDatasetImpl,TestFsVolumeList,TestInterDatanodeProtocol,TestLazyPersistFiles,TestRbwSpaceReservation,TestReplicaMap,TestScrLazyPersistFiles,TestWriteToReplica"
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707746/HDFS-7999-001.patch
          against trunk revision af618f2.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancer

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10089//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10089//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707746/HDFS-7999-001.patch against trunk revision af618f2. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancer The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10089//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10089//console This message is automatically generated.
          Hide
          xinwei Xinwei Qin added a comment -

          Hi, zhouyingchao
          Your idea is very good, but I don't think it can solve the problem you met fundamentally, especially on hundreds of threads writing concurrently. Because the heartbeat may be can not compete with other hundreds of createTemporary() threads successfully to get the dataset lock.

          Actually, I think we can just remove the dataset lock of heartbeat. The details can refer to https://issues.apache.org/jira/browse/HDFS-7060.

          Do you and others have any ideas?

          Show
          xinwei Xinwei Qin added a comment - Hi, zhouyingchao Your idea is very good, but I don't think it can solve the problem you met fundamentally, especially on hundreds of threads writing concurrently. Because the heartbeat may be can not compete with other hundreds of createTemporary() threads successfully to get the dataset lock. Actually, I think we can just remove the dataset lock of heartbeat. The details can refer to https://issues.apache.org/jira/browse/HDFS-7060 . Do you and others have any ideas?
          Hide
          sinago zhouyingchao added a comment -

          Hi Xinwei
          Thank you for sharing the status regarding HDFS-7060. I think it is the right way to fix the heartbeat issue. Saying that, I still think the patch here is necessary - current implementation of createTemporary() might sleep up to 60s with a lock held, it does not make sense, right? It might block other threads besides heartbeat for a long time any way.
          Comments? Thoughts?

          Show
          sinago zhouyingchao added a comment - Hi Xinwei Thank you for sharing the status regarding HDFS-7060 . I think it is the right way to fix the heartbeat issue. Saying that, I still think the patch here is necessary - current implementation of createTemporary() might sleep up to 60s with a lock held, it does not make sense, right? It might block other threads besides heartbeat for a long time any way. Comments? Thoughts?
          Hide
          sinago zhouyingchao added a comment -

          I tested TestBalancer with the patch on my rig, it can pass. From the log of the failure, it looks like not related to the patch.

          Show
          sinago zhouyingchao added a comment - I tested TestBalancer with the patch on my rig, it can pass. From the log of the failure, it looks like not related to the patch.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Xinwei Qin , even if we made the heartbeat lockless, there are still many other problems associated with having FsDatasetImpl#createTemporary hold the FSDatasetImpl lock for a very long time. Any thread that needs to read or write from the datanode will be blocked.

          I might be missing something (I'm not that familiar with this code), but I'm having trouble understanding the patch. Isn't it going to throw this exception:

          1447	            throw new ReplicaAlreadyExistsException("Block " + b
          1448	                + " already exists in state " + currentReplicaInfo.getState()
          1449	                + " and thus cannot be created.");
          

          whereas previously it would have called ReplicaInfo#stopWriter? Is that behavior change really intended?

          It seems like what we should have is a non-blocking version of stopWriter, as well as a loop that waits for the writer to stop.

          Also what is the significance of checking if we can get the same result from VolumeMap#get twice in a row? Why break out of the loop when that happens? There also seem to be some potential null pointer dereferences.

          Show
          cmccabe Colin P. McCabe added a comment - Xinwei Qin , even if we made the heartbeat lockless, there are still many other problems associated with having FsDatasetImpl#createTemporary hold the FSDatasetImpl lock for a very long time. Any thread that needs to read or write from the datanode will be blocked. I might be missing something (I'm not that familiar with this code), but I'm having trouble understanding the patch. Isn't it going to throw this exception: 1447 throw new ReplicaAlreadyExistsException( "Block " + b 1448 + " already exists in state " + currentReplicaInfo.getState() 1449 + " and thus cannot be created." ); whereas previously it would have called ReplicaInfo#stopWriter ? Is that behavior change really intended? It seems like what we should have is a non-blocking version of stopWriter , as well as a loop that waits for the writer to stop. Also what is the significance of checking if we can get the same result from VolumeMap#get twice in a row? Why break out of the loop when that happens? There also seem to be some potential null pointer dereferences.
          Hide
          sinago zhouyingchao added a comment -

          Thank you for looking into the patch. Here is some explain of the logic of createTemporary() after applying the patch:
          1. If there is no ReplicaInfo in volumeMap for the passed in ExtendedBlock b, then we will create one, insert into volumeMap and then return from line 1443.
          2. If there is a ReplicaInfo in volumeMap and its GS is newer than the passed in ExtendedBlock b, then throw the ReplicaAlreadyExistsException from line 1447.
          3. If there is a ReplicaInfo in volumeMap whereas its GS is older than the passed in ExtendedBlock b, then it means this is a new write and the earlier writer should be stopped. We will release the FsDatasetImpl lock and try to stop the earlier writer w/o the lock.
          4. After the earlier writer is stopped, we need to evict earlier writer's ReplicaInfo from volumeMap, to that end we will re-acquire the FsDatasetImpl lock. However, since this thread has released the FsDatasetImpl lock when it tried to stop earlier writer, another thread might have come in and changed the ReplicaInfo of this block in VolumeMap. This situation is not very likely to happen whereas we have to handle it in case. The loop in the patch is just tried to handle this situation – after re-acuire the FsDatasetImpl lock, it will check if the current ReplicaInfo in volumeMap is still the one before we stop the writer, if so we can simply evict it and create/insert a new one then return from line 1443. Otherwise, it implies another thread has slipped in and changed the ReplicaInfo when we were stopping earlier writer. In this condition, we check if that thread has inserted a block with even newer GS than us, if so we throws ReplicaAlreadyExistsException from line 1447. Otherwise we need to stop that thread's write just like we stop the earlier writer in step 3.

          Show
          sinago zhouyingchao added a comment - Thank you for looking into the patch. Here is some explain of the logic of createTemporary() after applying the patch: 1. If there is no ReplicaInfo in volumeMap for the passed in ExtendedBlock b, then we will create one, insert into volumeMap and then return from line 1443. 2. If there is a ReplicaInfo in volumeMap and its GS is newer than the passed in ExtendedBlock b, then throw the ReplicaAlreadyExistsException from line 1447. 3. If there is a ReplicaInfo in volumeMap whereas its GS is older than the passed in ExtendedBlock b, then it means this is a new write and the earlier writer should be stopped. We will release the FsDatasetImpl lock and try to stop the earlier writer w/o the lock. 4. After the earlier writer is stopped, we need to evict earlier writer's ReplicaInfo from volumeMap, to that end we will re-acquire the FsDatasetImpl lock. However, since this thread has released the FsDatasetImpl lock when it tried to stop earlier writer, another thread might have come in and changed the ReplicaInfo of this block in VolumeMap. This situation is not very likely to happen whereas we have to handle it in case. The loop in the patch is just tried to handle this situation – after re-acuire the FsDatasetImpl lock, it will check if the current ReplicaInfo in volumeMap is still the one before we stop the writer, if so we can simply evict it and create/insert a new one then return from line 1443. Otherwise, it implies another thread has slipped in and changed the ReplicaInfo when we were stopping earlier writer. In this condition, we check if that thread has inserted a block with even newer GS than us, if so we throws ReplicaAlreadyExistsException from line 1447. Otherwise we need to stop that thread's write just like we stop the earlier writer in step 3.
          Hide
          xinwei Xinwei Qin added a comment -

          Yeah, It's a good and necessary idea to avoid holding the lock for a long time by the createTemporary() method.

          Show
          xinwei Xinwei Qin added a comment - Yeah, It's a good and necessary idea to avoid holding the lock for a long time by the createTemporary() method.
          Hide
          xinwei Xinwei Qin added a comment -

          Hi Colin P. McCabe
          Thanks for your comment.

          even if we made the heartbeat lockless, there are still many other problems associated with having FsDatasetImpl#createTemporary hold the FSDatasetImpl lock for a very long time. Any thread that needs to read or write from the datanode will be blocked.

          Make the heartbeat lockless can avoid the happening of dead DataNode, and I think it is a necessary patch(https://issues.apache.org/jira/browse/HDFS-7060).
          FSDatasetImpl lock held for a long time is another problem, May be the patch of this jira can alleviate the problem.

          Show
          xinwei Xinwei Qin added a comment - Hi Colin P. McCabe Thanks for your comment. even if we made the heartbeat lockless, there are still many other problems associated with having FsDatasetImpl#createTemporary hold the FSDatasetImpl lock for a very long time. Any thread that needs to read or write from the datanode will be blocked. Make the heartbeat lockless can avoid the happening of dead DataNode, and I think it is a necessary patch( https://issues.apache.org/jira/browse/HDFS-7060 ). FSDatasetImpl lock held for a long time is another problem, May be the patch of this jira can alleviate the problem.
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          Thanks for the explanation¸ zhouyingchao. The patch makes sense.

          1455	      // Hang too long, just bail out. This is not supposed to happen.
          1456	      if (Time.monotonicNow() - startTime > bailOutDuration) {
          1457	        break;
          1458	      }
          

          Can you throw an exception from here rather than breaking? It seems like it would be clearer. Also, please log a WARN message to explain that there has been a problem. I would prefer to just see a log message rather than a comment explaining that "this is not supposed to happen"

          Instead of naming the timeout period "bailOutDuration", how about something like "writerStopTimeoutMs". In general, timeouts that are in milliseconds should end in ms.

          1464    throw new IOException("Hang " + ((Time.monotonicNow() - startTime) / 1000)
          1465	        + " seconds in createTemporary, just bail out");
          

          This error message seems confusing. It should be something like "Unable to stop existing writer for $REPLICA after $WHATEVER milliseconds."

          I think it looks good aside from that.

          Xinwei Qin wrote: Make the heartbeat lockless can avoid the happening of dead DataNode, and I think it is a necessary

          I think it is a good idea to make the heartbeat lockless. However, it is an exaggeration to say that it is necessary. The heartbeat wasn't lockless in previous releases of Hadoop such as 2.1, 2.3, or 2.5 and there were no complaints.

          Show
          cmccabe Colin P. McCabe added a comment - - edited Thanks for the explanation¸ zhouyingchao . The patch makes sense. 1455 // Hang too long , just bail out. This is not supposed to happen. 1456 if (Time.monotonicNow() - startTime > bailOutDuration) { 1457 break ; 1458 } Can you throw an exception from here rather than breaking? It seems like it would be clearer. Also, please log a WARN message to explain that there has been a problem. I would prefer to just see a log message rather than a comment explaining that "this is not supposed to happen" Instead of naming the timeout period "bailOutDuration", how about something like "writerStopTimeoutMs". In general, timeouts that are in milliseconds should end in ms. 1464 throw new IOException( "Hang " + ((Time.monotonicNow() - startTime) / 1000) 1465 + " seconds in createTemporary, just bail out" ); This error message seems confusing. It should be something like "Unable to stop existing writer for $REPLICA after $WHATEVER milliseconds." I think it looks good aside from that. Xinwei Qin wrote: Make the heartbeat lockless can avoid the happening of dead DataNode, and I think it is a necessary I think it is a good idea to make the heartbeat lockless. However, it is an exaggeration to say that it is necessary. The heartbeat wasn't lockless in previous releases of Hadoop such as 2.1, 2.3, or 2.5 and there were no complaints.
          Hide
          sinago zhouyingchao added a comment -

          Thanks a lot for the comments, Colin. I would update the patch accordingly.

          Show
          sinago zhouyingchao added a comment - Thanks a lot for the comments, Colin. I would update the patch accordingly.
          Hide
          sinago zhouyingchao added a comment -

          Test with "-Dtest=FsDatasetTestUtil,LazyPersistTestCase,TestDatanodeRestart,TestFsDatasetImpl,TestFsVolumeList,TestInterDatanodeProtocol,TestLazyPersistFiles,TestRbwSpaceReservation,TestReplicaMap,TestScrLazyPersistFiles,TestWriteToReplica,TestBalancer,TestDatanodeManager

          Show
          sinago zhouyingchao added a comment - Test with "-Dtest=FsDatasetTestUtil,LazyPersistTestCase,TestDatanodeRestart,TestFsDatasetImpl,TestFsVolumeList,TestInterDatanodeProtocol,TestLazyPersistFiles,TestRbwSpaceReservation,TestReplicaMap,TestScrLazyPersistFiles,TestWriteToReplica,TestBalancer,TestDatanodeManager
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12708958/HDFS-7999-002.patch
          against trunk revision 867d5d2.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10161//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10161//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12708958/HDFS-7999-002.patch against trunk revision 867d5d2. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10161//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10161//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -
          1458	        LOG.warn("Fail to stop existing writer for block " + b + " in "
          1459	            + writerStopMs + " miniseconds.");
          1460	        throw new IOException("Unable to stop existing writer for block " + b
          1461	            + " after " + writerStopMs + " miniseconds.");
          

          Can we have these both match up? I guess both should say "Unable to stop existing writer for block..."

          +1 once that's addressed. Thanks, zhouyingchao.

          Show
          cmccabe Colin P. McCabe added a comment - 1458 LOG.warn( "Fail to stop existing writer for block " + b + " in " 1459 + writerStopMs + " miniseconds." ); 1460 throw new IOException( "Unable to stop existing writer for block " + b 1461 + " after " + writerStopMs + " miniseconds." ); Can we have these both match up? I guess both should say "Unable to stop existing writer for block..." +1 once that's addressed. Thanks, zhouyingchao .
          Hide
          sinago zhouyingchao added a comment -

          Thank you, Colin. I'll update the patch soon.

          Show
          sinago zhouyingchao added a comment - Thank you, Colin. I'll update the patch soon.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12709424/HDFS-7999-003.patch
          against trunk revision ef591b1.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10179//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10179//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12709424/HDFS-7999-003.patch against trunk revision ef591b1. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10179//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10179//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1. Thnaks, zhouyingchao.

          Show
          cmccabe Colin P. McCabe added a comment - +1. Thnaks, zhouyingchao .
          Hide
          cmccabe Colin P. McCabe added a comment -

          Committed to 2.7. Thanks, guys.

          Show
          cmccabe Colin P. McCabe added a comment - Committed to 2.7. Thanks, guys.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7515 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7515/)
          HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7515 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7515/ ) HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #156 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/156/)
          HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #156 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/156/ ) HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #890 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/890/)
          HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #890 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/890/ ) HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #147 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/147/)
          HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #147 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/147/ ) HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2088/)
          HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2088/ ) HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/157/)
          HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/157/ ) HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2106 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2106/)
          HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2106 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2106/ ) HDFS-7999 . FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) (cmccabe: rev 28bebc81db8bb6d1bc2574de7564fe4c595cfe09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Sangjin Lee backported this to 2.6.1, after fixing some non-trivial merge conflicts.

          I just pushed the commit to 2.6.1 after running compilation.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Sangjin Lee backported this to 2.6.1, after fixing some non-trivial merge conflicts. I just pushed the commit to 2.6.1 after running compilation.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Attaching the patch for 2.6.1 that I committed.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Attaching the patch for 2.6.1 that I committed.

            People

            • Assignee:
              sinago zhouyingchao
              Reporter:
              sinago zhouyingchao
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development