Thu Jul 17 17:04:25 UTC 2008 Starting wtp.back.cluster.HRegionServer on sjdc-atr-test-4 java version "1.6.0_03" Java(TM) SE Runtime Environment (build 1.6.0_03-b05) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_03-b05, mixed mode) ulimit -n 8192 2008-07-17 17:04:26,224 INFO wtp.back.cluster.HRegionServer: looking for dfs:namenode 2008-07-17 17:04:26,373 INFO wtp.back.cluster.HRegionServer: found service://dfs/namenode at dfs://sjdc-atr-dns.atr.trendmicro.com:50000/?namenode 2008-07-17 17:04:26,373 INFO wtp.back.cluster.HRegionServer: looking for hbase:master 2008-07-17 17:04:26,457 INFO wtp.back.cluster.HRegionServer: found service://hbase/master at hbase://sjdc-atr-dns.atr.trendmicro.com:60000/?master 2008-07-17 17:04:26,460 INFO wtp.back.cluster.Node: registered service service://hbase/regionserver at hbase://sjdc-atr-test-4.atr.trendmicro.com:60020/?regionserver 2008-07-17 17:04:26,643 INFO wtp.back.cluster.Node: wrote /tmp/wtp.back.cluster.HMaster.sjdc-atr-test-4.atr.trendmicro.com.60020/hadoop-site.xml 2008-07-17 17:04:26,709 INFO wtp.back.cluster.Node: wrote /tmp/wtp.back.cluster.HMaster.sjdc-atr-test-4.atr.trendmicro.com.60020/hbase-site.xml 2008-07-17 17:04:27,472 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=60020, port=60020 2008-07-17 17:04:27,814 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216314267787 2008-07-17 17:04:27,860 INFO org.mortbay.util.Credential: Checking Resource aliases 2008-07-17 17:04:27,889 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4 2008-07-17 17:04:27,889 INFO org.mortbay.util.Container: Started HttpContext[/static,/static] 2008-07-17 17:04:27,889 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs] 2008-07-17 17:04:28,382 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@3c1fc1a6 2008-07-17 17:04:28,413 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/] 2008-07-17 17:04:28,415 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60030 2008-07-17 17:04:28,415 INFO org.mortbay.util.Container: Started org.mortbay.jetty.Server@5c435a3a 2008-07-17 17:04:28,415 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2008-07-17 17:04:28,422 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020: starting 2008-07-17 17:04:28,422 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60020: starting 2008-07-17 17:04:28,422 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020: starting 2008-07-17 17:04:28,422 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020: starting 2008-07-17 17:04:28,422 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020: starting 2008-07-17 17:04:28,423 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020: starting 2008-07-17 17:04:28,423 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020: starting 2008-07-17 17:04:28,423 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020: starting 2008-07-17 17:04:28,423 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020: starting 2008-07-17 17:04:28,423 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020: starting 2008-07-17 17:04:28,423 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started at: 10.30.94.6:60020 2008-07-17 17:04:28,424 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020: starting 2008-07-17 17:34:28,355 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of entries: 0 2008-07-17 17:34:28,382 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216316068378 2008-07-17 17:34:28,383 INFO org.apache.hadoop.hbase.regionserver.HLog: removing old log file /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.0 whose highest sequence/edit id is -1 2008-07-17 17:42:07,393 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: t1,,1216314329733 2008-07-17 17:42:07,395 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: t1,,1216314329733 2008-07-17 17:42:08,860 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2008-07-17 17:42:08,862 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 2008-07-17 17:42:09,103 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t1,,1216314329733/1363430436 available 2008-07-17 17:42:09,104 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region t1,,1216314329733 2008-07-17 17:42:09,108 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region t1,,1216314329733 in 0sec 2008-07-17 17:46:19,693 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of entries: 31000 2008-07-17 17:46:19,706 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216316779703 2008-07-17 17:50:15,720 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of entries: 31000 2008-07-17 17:50:15,734 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216317015731 2008-07-17 17:54:25,664 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of entries: 31000 2008-07-17 17:54:25,677 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216317265674 2008-07-17 17:58:49,299 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of entries: 31000 2008-07-17 17:58:49,319 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216317529315 2008-07-17 18:03:22,766 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of entries: 31000 2008-07-17 18:03:22,781 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216317802778 2008-07-17 18:04:33,458 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of entries: 7000 2008-07-17 18:04:33,470 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216317873467 2008-07-17 18:09:17,999 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of entries: 31000 2008-07-17 18:09:18,012 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer created at /data/hadoop/hbase/log_10.30.94.6_1216314267517_60020/hlog.dat.1216318158009 2008-07-17 18:12:18,051 FATAL org.apache.hadoop.hbase.regionserver.Flusher: Replay of hlog required. Forcing server restart org.apache.hadoop.hbase.DroppedSnapshotException at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1040) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:942) at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174) at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:93) Caused by: java.lang.NullPointerException at java.lang.String.(String.java:516) at org.apache.hadoop.hbase.util.Bytes.toString(Bytes.java:71) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.updateReaders(StoreFileScanner.java:374) at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:797) at org.apache.hadoop.hbase.regionserver.HStore.updateReaders(HStore.java:784) at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:755) at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:682) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1030) ... 3 more 2008-07-17 18:12:18,052 INFO org.apache.hadoop.hbase.regionserver.Flusher: regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting 2008-07-17 18:12:18,161 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting. 2008-07-17 18:12:19,090 INFO org.apache.hadoop.ipc.Server: Stopping server on 60020 2008-07-17 18:12:19,090 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020: exiting 2008-07-17 18:12:19,090 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020: exiting 2008-07-17 18:12:19,090 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020: exiting 2008-07-17 18:12:19,090 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020: exiting 2008-07-17 18:12:19,090 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020: exiting 2008-07-17 18:12:19,090 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020: exiting 2008-07-17 18:12:19,090 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60020 2008-07-17 18:12:19,091 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020: exiting 2008-07-17 18:12:19,091 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer 2008-07-17 18:12:19,091 INFO org.mortbay.util.ThreadedServer: Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030] 2008-07-17 18:12:19,091 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2008-07-17 18:12:19,093 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020: exiting 2008-07-17 18:12:19,093 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020: exiting 2008-07-17 18:12:19,275 INFO org.mortbay.http.SocketListener: Stopped SocketListener on 0.0.0.0:60030 2008-07-17 18:12:20,170 INFO org.mortbay.util.Container: Stopped HttpContext[/static,/static] 2008-07-17 18:12:20,684 INFO org.mortbay.util.Container: Stopped HttpContext[/logs,/logs] 2008-07-17 18:12:20,688 INFO org.mortbay.util.Container: Stopped org.mortbay.jetty.servlet.WebApplicationHandler@3c1fc1a6 2008-07-17 18:12:21,201 INFO org.mortbay.util.Container: Stopped WebApplicationContext[/,/] 2008-07-17 18:12:21,201 INFO org.mortbay.util.Container: Stopped org.mortbay.jetty.Server@5c435a3a 2008-07-17 18:12:21,211 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting 2008-07-17 18:12:21,263 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed hlog 2008-07-17 18:12:21,645 WARN org.apache.hadoop.dfs.DFSClient: Exception while reading from blk_-5056743456606246938 of /data/hadoop/hbase/t1/1363430436/c1/mapfiles/4968729745733087265/data from 10.30.94.6:50010: java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:54) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:149) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:98) at org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967) at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236) at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:178) at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:195) at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159) at org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:829) at org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1352) at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1388) at java.io.DataInputStream.readFully(DataInputStream.java:178) at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1882) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1782) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1828) at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:516) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.getNext(StoreFileScanner.java:306) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180) at org.apache.hadoop.hbase.regionserver.HStoreScanner.next(HStoreScanner.java:194) at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1786) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1293) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1186) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) 2008-07-17 18:12:21,645 INFO org.apache.hadoop.dfs.DFSClient: Could not obtain block blk_-5056743456606246938 from any node: java.io.IOException: No live nodes contain current block 2008-07-17 18:12:24,107 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting 2008-07-17 18:12:24,107 INFO org.apache.hadoop.hbase.Leases: regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closing leases 2008-07-17 18:12:24,108 INFO org.apache.hadoop.hbase.Leases: regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closed leases 2008-07-17 18:12:26,087 WARN org.apache.hadoop.ipc.Server: IPC Server Responder, call openScanner([B@7d8ab25b, [[B@1cc3b4c1, [B@748ff9a7, 9223372036854775807, null) from 10.30.94.1:48602: output error 2008-07-17 18:12:26,087 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020 caught: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:594) at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:654) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:917) 2008-07-17 18:12:26,087 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020: exiting