2011-08-25 20:49:26,462 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 136428ms for sessionid 0x131ed2399c30075, closing socket connection and attempting reconnect 2011-08-25 20:49:26,504 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog java.io.IOException: Reflection at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147) at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986) at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:963) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145) ... 2 more Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$Packet.(DFSClient.java:2313) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:3220) at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3307) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97) at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944) ... 6 more 2011-08-25 20:49:26,462 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39) at java.nio.ByteBuffer.allocate(ByteBuffer.java:312) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1093) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1276) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:87) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:82) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:297) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2350) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2306) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2323) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1823) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) 2011-08-25 20:49:34,951 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call next(9114966683188142382, 30) from 192.168.41.3:53384: output error 2011-08-25 20:49:34,947 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_3784810663390554197_157754572java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.41.3:42855 remote=/192.168.41.3:50010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.DataInputStream.readFully(DataInputStream.java:178) at java.io.DataInputStream.readLong(DataInputStream.java:399) at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:122) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2601)