1.In order to tracing what happened, I added some logs: public byte [][] rollWriter() throws FailedLogCloseException, IOException { // Return if nothing to flush. long stamp = System.currentTimeMillis(); if (this.writer != null && this.numEntries.get() <= 0) { LOG.info("[jeasonlogs]rollWriter: return directly.." + stamp); return null; } byte [][] regionsToFlush = null; LOG.info("[jeasonlogs]rollWriter:Current rollWriter thread is : " + Thread.currentThread() + " stamp: " + stamp); LOG.info("[jeasonlogs]rollWriter: lock owner is : " + this.cacheFlushLock.getOwner() + " stamp: " + stamp); this.cacheFlushLock.lock(); ------ } @Override public void sync() throws IOException { ------- try { -------- synchronized (this.updateLock) { syncTime += System.currentTimeMillis() - now; syncOps++; LOG.info("[jeasonlogs]sync: lock owner is : " + this.cacheFlushLock.getOwner() + " logRollRequeted: " + logRollRequested); -------- } } catch (IOException e) { ------- } } 2. The related logs: 2011-07-28 12:21:08,370 INFO org.apache.hadoop.hdfs.DFSClient: Excluding datanode 167.6.1.12:10010 2011-07-28 12:21:08,390 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : null logRollRequeted: false 2011-07-28 12:21:08,431 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas. Requesting close of hlog. 2011-07-28 12:21:08,431 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : null logRollRequeted: true 2011-07-28 12:21:08,432 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : null logRollRequeted: true 2011-07-28 12:21:08,432 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]rollWriter:Current rollWriter thread is : Thread[regionserver20020.logRoller,5,main] stamp: 1311826868431 2011-07-28 12:21:08,432 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]rollWriter: lock owner is : null stamp: 1311826868431 2011-07-28 12:21:08,432 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,433 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,434 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,450 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,458 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,469 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,490 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,507 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,520 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,531 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,554 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,563 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,573 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,582 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,623 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,623 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,624 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,624 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,625 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,771 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : Thread[regionserver20020.logRoller,5,main] logRollRequeted: true 2011-07-28 12:21:08,810 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200 2011-07-28 12:21:08,845 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/167-6-1-11,20020,1311826019178/167-6-1-11%3A20020.1311826868295, entries=23, filesize=1132442. New hlog /hbase/.logs/167-6-1-11,20020,1311826019178/167-6-1-11%3A20020.1311826868432 stamp: 1311826868431 2011-07-28 12:21:08,908 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream Bad connect ack with firstBadLink 167.6.1.12:10010 2011-07-28 12:21:08,908 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_1311732886686_8050 2011-07-28 12:21:08,908 INFO org.apache.hadoop.hdfs.DFSClient: Excluding datanode 167.6.1.12:10010 2011-07-28 12:21:08,944 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : null logRollRequeted: false 2011-07-28 12:21:08,944 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas. Requesting close of hlog. 2011-07-28 12:21:08,945 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]rollWriter: return directly..1311826868945 2011-07-28 12:21:08,952 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : null logRollRequeted: true 2011-07-28 12:21:08,954 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : null logRollRequeted: true 2011-07-28 12:21:08,972 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: [jeasonlogs]sync: lock owner is : null logRollRequeted: true