HBase
  1. HBase
  2. HBASE-15

[hbase] Could not complete hdfs write out to flush file forcing regionserver restart

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None

      Description

      I've spent some time looking into this issue but there are not enough clues in the logs to tell where the problem is. Here's what I know.

      Two region servers went down last night, a minute apart, during Paul Saab's 6hr run inserting 300million rows into hbase. The regionservers went down to force rerun of hlog and avoid possible data loss after a failure writing memory flushes to hdfs.

      Here is the lead up to the failed flush:

      ...
      2007-11-28 22:40:02,231 INFO hbase.HRegionServer - MSG_REGION_OPEN : regionname: postlog,img149/4699/133lm0.jpg,1196318393738, startKey: <img149/4699/133lm0.jpg>, tableDesc: {name: postlog, families: {cookie:=

      {name: cookie, max versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}

      , ip:=

      {name: ip, max versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}

      }}
      2007-11-28 22:40:02,242 DEBUG hbase.HStore - starting 1703405830/cookie (no reconstruction log)
      2007-11-28 22:40:02,741 DEBUG hbase.HStore - maximum sequence id for hstore 1703405830/cookie is 29077708
      2007-11-28 22:40:03,094 DEBUG hbase.HStore - starting 1703405830/ip (no reconstruction log)
      2007-11-28 22:40:03,852 DEBUG hbase.HStore - maximum sequence id for hstore 1703405830/ip is 29077708
      2007-11-28 22:40:04,138 DEBUG hbase.HRegion - Next sequence id for region postlog,img149/4699/133lm0.jpg,1196318393738 is 29077709
      2007-11-28 22:40:04,141 INFO hbase.HRegion - region postlog,img149/4699/133lm0.jpg,1196318393738 available
      2007-11-28 22:40:04,141 DEBUG hbase.HLog - changing sequence number from 21357623 to 29077709
      2007-11-28 22:40:04,141 INFO hbase.HRegionServer - MSG_REGION_OPEN : regionname: postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739, startKey: <img149/7512/dscn4444lightenedfi3.jpg>, tableDesc: {name: postlog, families: {cookie:=

      {name: cookie, max versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}

      , ip:=

      {name: ip, max versions: 1, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}

      }}
      2007-11-28 22:40:04,145 DEBUG hbase.HStore - starting 376748222/cookie (no reconstruction log)
      2007-11-28 22:40:04,223 DEBUG hbase.HStore - maximum sequence id for hstore 376748222/cookie is 29077708
      2007-11-28 22:40:04,277 DEBUG hbase.HStore - starting 376748222/ip (no reconstruction log)
      2007-11-28 22:40:04,353 DEBUG hbase.HStore - maximum sequence id for hstore 376748222/ip is 29077708
      2007-11-28 22:40:04,699 DEBUG hbase.HRegion - Next sequence id for region postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739 is 29077709
      2007-11-28 22:40:04,701 INFO hbase.HRegion - region postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739 available
      2007-11-28 22:40:34,427 DEBUG hbase.HRegionServer - flushing region postlog,img143/1310/yashrk3.jpg,1196317258704
      2007-11-28 22:40:34,428 DEBUG hbase.HRegion - Not flushing cache for region postlog,img143/1310/yashrk3.jpg,1196317258704: snapshotMemcaches() determined that there was nothing to do
      2007-11-28 22:40:55,745 DEBUG hbase.HRegionServer - flushing region postlog,img142/8773/1001417zc4.jpg,1196317258703
      2007-11-28 22:40:55,745 DEBUG hbase.HRegion - Not flushing cache for region postlog,img142/8773/1001417zc4.jpg,1196317258703: snapshotMemcaches() determined that there was nothing to do
      2007-11-28 22:41:04,144 DEBUG hbase.HRegionServer - flushing region postlog,img149/4699/133lm0.jpg,1196318393738
      2007-11-28 22:41:04,144 DEBUG hbase.HRegion - Started memcache flush for region postlog,img149/4699/133lm0.jpg,1196318393738. Size 74.7k
      2007-11-28 22:41:04,764 DEBUG hbase.HStore - Added 1703405830/ip/610047924323344967 with sequence id 29081563 and size 53.8k
      2007-11-28 22:41:04,902 DEBUG hbase.HStore - Added 1703405830/cookie/3147798053949544972 with sequence id 29081563 and size 41.3k
      2007-11-28 22:41:04,902 DEBUG hbase.HRegion - Finished memcache flush for region postlog,img149/4699/133lm0.jpg,1196318393738 in 758ms, sequenceid=29081563
      2007-11-28 22:41:04,902 DEBUG hbase.HStore - compaction for HStore postlog,img149/4699/133lm0.jpg,1196318393738/ip needed.
      2007-11-28 22:41:04,903 DEBUG hbase.HRegion - 1703405830/ip needs compaction
      2007-11-28 22:41:04,903 INFO hbase.HRegion - starting compaction on region postlog,img149/4699/133lm0.jpg,1196318393738
      2007-11-28 22:41:04,903 DEBUG hbase.HStore - started compaction of 4 files in /hbase/compaction.dir/hregion_1703405830/ip
      2007-11-28 22:41:04,905 DEBUG hbase.HRegionServer - flushing region postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739
      2007-11-28 22:41:04,906 DEBUG hbase.HRegion - Started memcache flush for region postlog,img149/7512/dscn4444lightenedfi3.jpg,1196318393739. Size 133.1k
      2007-11-28 22:41:05,087 DEBUG hbase.HStore - Added 376748222/ip/8686640221458382286 with sequence id 29081604 and size 95.8k
      2007-11-28 22:41:05,419 FATAL hbase.HRegionServer - Replay of hlog required. Forcing server restart
      org.apache.hadoop.hbase.DroppedSnapshotException: java.io.IOException: Could not complete write to file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data by DFSClient_-1010714652
      at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:323)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

      at org.apache.hadoop.hbase.HRegion.internalFlushcache(HRegion.java:920)
      at org.apache.hadoop.hbase.HRegion.flushcache(HRegion.java:807)
      at org.apache.hadoop.hbase.HRegionServer$Flusher.run(HRegionServer.java:451)
      ...

      Over in the namenode, I see the following related to the deleted file:

      2007-11-28 22:40:48,546 DEBUG dfs.StateChange - DIR NameNode.mkdirs: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
      2007-11-28 22:40:48,546 DEBUG dfs.StateChange - DIR* NameSystem.mkdirs: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
      2007-11-28 22:40:48,546 DEBUG dfs.StateChange - DIR* FSDirectory.mkdirs: created directory /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
      2007-11-28 22:40:48,552 DEBUG dfs.StateChange - DIR NameNode.create: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652 at XX.XX.XX.29
      2007-11-28 22:40:48,552 DEBUG dfs.StateChange - DIR* NameSystem.startFile: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652 at XX.XX.XX.29
      2007-11-28 22:40:48,552 DEBUG dfs.StateChange - DIR* FSDirectory.addFile: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data is added to the file system
      2007-11-28 22:40:48,552 DEBUG dfs.StateChange - DIR* NameSystem.startFile: add /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data to namespace for DFSClient_-1010714652
      2007-11-28 22:40:48,557 DEBUG dfs.StateChange - DIR NameNode.create: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/index for DFSClient_-1010714652 at XX.XX.XX.29
      2007-11-28 22:40:48,557 DEBUG dfs.StateChange - DIR* NameSystem.startFile: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/index for DFSClient_-1010714652 XX.XX.XX.29
      2007-11-28 22:40:48,558 DEBUG dfs.StateChange - DIR* FSDirectory.addFile: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/index is added to the file system
      2007-11-28 22:40:48,558 DEBUG dfs.StateChange - DIR* NameSystem.startFile: add /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/index to namespace for DFSClient_-1010714652
      2007-11-28 22:40:48,573 DEBUG dfs.StateChange - BLOCK NameNode.addBlock: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652
      2007-11-28 22:40:48,573 DEBUG dfs.StateChange - BLOCK* NameSystem.getAdditionalBlock: file /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652
      2007-11-28 22:40:48,573 DEBUG dfs.StateChange - DIR* FSDirectory.addFile: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data with blk_-6398846109350112398 block is added to the in-memory file system
      2007-11-28 22:40:48,573 INFO dfs.StateChange - BLOCK* NameSystem.allocateBlock: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data. blk_-6398846109350112398
      .....
      2007-11-28 22:40:48,717 DEBUG dfs.StateChange - DIR NameNode.delete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
      2007-11-28 22:40:48,717 DEBUG dfs.StateChange - DIR* NameSystem.delete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
      2007-11-28 22:40:48,717 DEBUG dfs.StateChange - DIR* FSDirectory.delete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411
      2007-11-28 22:40:48,717 DEBUG dfs.StateChange - DIR* FSDirectory.unprotectedDelete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411 is removed
      ....
      2007-11-28 22:40:48,826 DEBUG dfs.StateChange - DIR NameNode.complete: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652
      2007-11-28 22:40:48,826 DEBUG dfs.StateChange - DIR* NameSystem.completeFile: /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data for DFSClient_-1010714652
      2007-11-28 22:40:48,827 WARN dfs.StateChange - DIR* NameSystem.completeFile: failed to complete /hbase/hregion_376748222/cookie/mapfiles/3557442410867103411/data because dir.getFileBlocks() is null and pendingFile is null

      Only delete I see is on compaction completion but reading code doesn't look like its possible for deletes to be confused.

      Will add extra logging to see if I can figure how the delete is coming about.

      (We've fixed a similar problem before when compactions were done at region level)

        Issue Links

          Activity

          Hide
          stack added a comment -

          Log all deletes.

          Show
          stack added a comment - Log all deletes.
          Hide
          Bryan Duxbury added a comment -

          Sounds serious. Changing to critical.

          Show
          Bryan Duxbury added a comment - Sounds serious. Changing to critical.
          Hide
          Jim Kellerman added a comment -

          Have we seen any more occurrences of this problem?

          If not should we close this issue as not reproducable and open a new one if it should happen again?

          Show
          Jim Kellerman added a comment - Have we seen any more occurrences of this problem? If not should we close this issue as not reproducable and open a new one if it should happen again?
          Hide
          Jim Kellerman added a comment -

          Dropping priority since this bug has not re-occurred.

          Show
          Jim Kellerman added a comment - Dropping priority since this bug has not re-occurred.
          Hide
          Andrew Purtell added a comment -

          Linked to HBASE-616. I think both issues are related to excessive I/O load, with processes stuck in disk wait, etc.

          Show
          Andrew Purtell added a comment - Linked to HBASE-616 . I think both issues are related to excessive I/O load, with processes stuck in disk wait, etc.
          Hide
          Andrew Purtell added a comment -

          From hbase-users@

          From: "Renaud Delbru" <renaud.delbru@deri.org>
          To: hbase-user@hadoop.apache.org
          Saturday, July 19, 2008 7:46 AM

          The cluster is currently composed of 3 nodes. The cluster is also used
          for executing hadoop jobs. One of the node acts as an hadoop master, an
          HMaster and an HRegionServer.
          The current settings are: Xmx=1024M for the Hadoop Master, Xmx=2048M for
          the HMaster, Xmx=2048M for each HRegionServer.

          The inserts are somehow randomised. We have 24 hadoop jobs uploading
          various documents. The row key is the URL of the document, and each
          hadoop task has to upload a range of URLs (for example,
          http://my-url/AAA - http://my-url/BBB).
          Thus, I assume that each hadoop tasks will hit a different region and
          that the insert load is distributed among the nodes.

          Thanks.

          Renaud Delbru

          Jean-Daniel Cryans wrote:
          > Ok then, at least having the nodes synchronized keeps you safe from other
          > problems. By the way, how many nodes do you have?
          >
          > One other thing to check is your writing pattern. If you insert data with
          > sequential row keys, an example would be designing your row keys as an
          > incremented primary key equivalent but left padded with zeros, these inserts
          > will always hit on a single region thus the load will not be distributed. If
          > it's your case, try randomizing the inserts.
          >
          > I will also look at your logs monday morning.
          >
          > Thx,
          >
          > J-D
          >
          > On Sat, Jul 19, 2008 at 5:47 AM, Renaud Delbru <renaud.delbru@deri.org>
          > wrote:
          >
          >
          >> Hi Jean-Daniel,
          >>
          >> I have synchronised all the nodes to the same ntp server yesterday.
          >> This morning, another regionserver went down for the same reason (log
          >> attached).
          >>
          >> Thanks.
          >> –
          >> Renaud Delbru
          >>
          >>
          >>
          >> Jean-Daniel Cryans wrote:
          >>
          >>
          >>> Renaud,
          >>>
          >>> Yes, the impact is important. Try fixing it first because typically heavy
          >>> write load problems only occur when there is a very few number of nodes
          >>> and
          >>> regions. We have seen this many times.
          >>>
          >>> Happy fixing!
          >>>
          >>> J-D
          >>>
          >>> On Fri, Jul 18, 2008 at 5:20 PM, Renaud Delbru <renaud.delbru@deri.org>
          >>> wrote:
          >>>
          >>>
          >>>
          >>>
          >>>> Hi Jean-Daniel,
          >>>>
          >>>> my nodes clocks are desynchronised (each node has a different time).
          >>>> Could
          >>>> it have an impact on HBase ?
          >>>>
          >>>> At the moment of the failure, the node had about 55 regions (with
          >>>> hbase.hregion.max.filesize = 128M).
          >>>>
          >>>> Thanks.
          >>>> –
          >>>> Renaud Delbru
          >>>>
          >>>> As One node is at
          >>>>
          >>>> Jean-Daniel Cryans wrote:
          >>>>
          >>>>
          >>>>
          >>>>
          >>>>> Renaud,
          >>>>>
          >>>>> Just to make sure, are all your nodes clocks synchronized? Also, at the
          >>>>> moment of that failure, how many regions did the table have?
          >>>>>
          >>>>> Thanks,
          >>>>>
          >>>>> J-D
          >>>>>
          >>>>> On Fri, Jul 18, 2008 at 5:11 AM, Renaud Delbru <renaud.delbru@deri.org>
          >>>>> wrote:
          >>>>>
          >>>>>
          >>>>>
          >>>>>
          >>>>>
          >>>>>
          >>>>>> Hi,
          >>>>>>
          >>>>>> I am using HBase 0.2.0-dev, Hudson Build #208.
          >>>>>> I am experiencing a problem with HBase that looks like the issue
          >>>>>> https://issues.apache.org/jira/browse/HBASE-15.
          >>>>>> During an upload intensive task, one of the regionserver went down to
          >>>>>> force rerun of hlog.
          >>>>>>
          >>>>>> This is not the first time we have experienced this problem, it
          >>>>>> happened
          >>>>>> already on another cluster node a few days ago, in the same settings.
          >>>>>>
          >>>>>> I have attached the regionserver log. I don't notice something
          >>>>>> particular in the hadoop datanode log.
          >>>>>>
          >>>>>> Thanks in advance for any advices.
          >>>>>> –
          >>>>>> Renaud Delbru
          >>>>>>
          >>>>>>
          >>>>>>
          >>>>>> 2008-07-18 03:40:51,861 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of
          >>>>>> entries: 30008
          >>>>>> 2008-07-18 03:40:52,073 INFO org.apache.hadoop.hbase.regionserver.HLog:
          >>>>>> New
          >>>>>> log writer created at
          >>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216348851959
          >>>>>> 2008-07-18 03:43:36,257 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of
          >>>>>> entries: 30008
          >>>>>> 2008-07-18 03:43:37,020 INFO org.apache.hadoop.hbase.regionserver.HLog:
          >>>>>> New
          >>>>>> log writer created at
          >>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216349016755
          >>>>>> 2008-07-18 03:45:12,963 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,12159192644012008-07-18
          >>>>>>
          >>>>>>> 03:45:58,868
          >>>>>>>
          >>>>>> WARN org.apache.hadoop.hbase.util.Sleeper: We slept 30837ms,
          >>>>>>
          >>>>>> ten times longer than scheduled: 3000
          >>>>>> 2008-07-18 03:55:57,783 WARN org.apache.hadoop.hbase.util.Sleeper: We
          >>>>>> slept
          >>>>>> 583701ms, ten times longer than scheduled: 3000
          >>>>>> 2008-07-18 03:55:57,787 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of
          >>>>>> entries: 21011
          >>>>>> 2008-07-18 03:55:58,410 WARN org.apache.hadoop.dfs.DFSClient: Exception
          >>>>>> while reading from blk_-6424749704096696952 of
          >>>>>>
          >>>>>> /hbase/page-repository/942413588/field/mapfiles/7570604533190594376/data
          >>>>>> from 140.203.154.176:50010: java.io.IOException: Premeture EOF from
          >>>>>> inputStream
          >>>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
          >>>>>> at
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
          >>>>>> at
          >>>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
          >>>>>> at
          >>>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
          >>>>>> at
          >>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:829)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1352)
          >>>>>> at
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1388)
          >>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:178)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1643)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:1682)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1799)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1828)
          >>>>>> at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:499)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.MapFileCompactionReader.next(MapFileCompactionReader.java:45)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStore.compactHStoreFiles(HStore.java:1029)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:896)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:880)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:840)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:84)
          >>>>>>
          >>>>>> 2008-07-18 03:55:58,966 WARN org.apache.hadoop.dfs.DFSClient: Exception
          >>>>>> while reading from blk_7305122468419183134 of
          >>>>>>
          >>>>>> /hbase/page-repository/942413588/field/mapfiles/4454700123966493614/data
          >>>>>> from 140.203.154.176:50010: java.io.IOException: Premeture EOF from
          >>>>>> inputStream
          >>>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
          >>>>>> at
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
          >>>>>> at
          >>>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
          >>>>>> at
          >>>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
          >>>>>> at
          >>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:829)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1352)
          >>>>>> at
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1388)
          >>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:178)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1643)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1709)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1745)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1831)
          >>>>>> at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:499)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.MapFileCompactionReader.next(MapFileCompactionReader.java:45)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStore.compactHStoreFiles(HStore.java:1029)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:896)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:880)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:840)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:84)
          >>>>>>
          >>>>>> 2008-07-18 03:56:05,305 INFO org.apache.hadoop.hbase.regionserver.HLog:
          >>>>>> New
          >>>>>> log writer created at
          >>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216349764848
          >>>>>> 2008-07-18 03:57:30,244 ERROR
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening
          >>>>>> scanner
          >>>>>> (fsOk: true)
          >>>>>> java.io.IOException: HStoreScanner failed construction
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:69)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:68)
          >>>>>> at
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1896)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1767)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1293)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1186)
          >>>>>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
          >>>>>> response
          >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:559)
          >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
          >>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
          >>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
          >>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
          >>>>>> at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:572)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:365)
          >>>>>> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:628)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.createDataFileReader(HStoreFile.java:677)
          >>>>>> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:659)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:644)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:734)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:409)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:95)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:66)
          >>>>>> ... 10 more
          >>>>>> 2008-07-18 03:58:01,895 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 7 on 60020, call openScanner([B@9dd0d92, [[B@fa5f14d,
          >>>>>> [B@5784fe41,
          >>>>>> 9223372036854775807, null) from 140.203.154.174:48958: error:
          >>>>>> java.io.IOException: HStoreScanner failed construction
          >>>>>> java.io.IOException: HStoreScanner failed construction
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:69)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:68)
          >>>>>> at
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1896)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1767)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1293)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1186)
          >>>>>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
          >>>>>> response
          >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:559)
          >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
          >>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
          >>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
          >>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
          >>>>>> at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:572)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:365)
          >>>>>> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:628)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.createDataFileReader(HStoreFile.java:677)
          >>>>>> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:659)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:644)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:734)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:409)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:95)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:66)
          >>>>>> ... 10 more
          >>>>>> 2008-07-18 03:58:06,728 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> compaction completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401 in 12mins,
          >>>>>> 53sec
          >>>>>> 2008-07-18 03:58:06,728 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Angus_McKay,1215890852482
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Angus_McKay,12158908524822008-07-18
          >>>>>>
          >>>>>>> 03:58:06,729
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Angus_McKay,1215890852482 in 0sec
          >>>>>> 2008-07-18 03:58:06,729 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,12159472244252008-07-18
          >>>>>>
          >>>>>>> 03:58:06,730
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425 in 0sec
          >>>>>> 2008-07-18 03:58:06,730 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,12159830915392008-07-18
          >>>>>>
          >>>>>>> 03:58:06,760
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539in0sec
          >>>>>> 2008-07-18 03:58:06,760 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,12145963998112008-07-18
          >>>>>>
          >>>>>>> 03:58:06,760
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811in0sec
          >>>>>> 2008-07-18 03:58:06,761 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,12159589559112008-07-18
          >>>>>>
          >>>>>>> 03:58:06,761
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911 in 0sec
          >>>>>> 2008-07-18 03:58:06,761 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,12158238964112008-07-18
          >>>>>>
          >>>>>>> 03:58:06,777
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411in0sec
          >>>>>> 2008-07-18 03:58:06,777 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,12158239811372008-07-18
          >>>>>>
          >>>>>>> 03:58:06,800
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137 in 0sec
          >>>>>> 2008-07-18 03:58:06,800 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,12160399606632008-07-18
          >>>>>>
          >>>>>>> 03:58:06,826
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663in
          >>>>>> 0sec
          >>>>>> 2008-07-18 03:58:06,826 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,12145475938142008-07-18
          >>>>>>
          >>>>>>> 03:58:06,916
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814in0sec
          >>>>>> 2008-07-18 03:58:06,916 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://sws.geonames.org/3025251/,1215824259058
          >>>>>> 2008-07-18 <http://sws.geonames.org/3025251/,12158242590582008-07-18
          >>>>>>
          >>>>>>> 03:58:06,972
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://sws.geonames.org/3025251/,1215824259058 in 0sec
          >>>>>> 2008-07-18 03:58:06,973 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,12161255630462008-07-18
          >>>>>>
          >>>>>>> 03:58:06,973
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046in
          >>>>>> 0sec
          >>>>>> 2008-07-18 03:58:06,973 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Flowers_of_Life,12160084595652008-07-18
          >>>>>>
          >>>>>>> 03:58:15,979
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling
          >>>>>> hlog. Number of entries: 30009
          >>>>>> 2008-07-18 03:59:32,093 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> compaction completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565 in 1mins,
          >>>>>> 25sec
          >>>>>> 2008-07-18 03:59:32,093 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330
          >>>>>> 2008-07-18<
          >>>>>> http://acm.rkbexplorer.com/id/person-592138,12158248783302008-07-18
          >>>>>>
          >>>>>>> 03:59:32,134
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330 in 0sec
          >>>>>> 2008-07-18 03:59:32,134 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,12159157700092008-07-18
          >>>>>>
          >>>>>>> 04:00:45,884
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009 in
          >>>>>> 1mins,
          >>>>>> 13sec
          >>>>>> 2008-07-18 04:00:45,885 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,12160445949432008-07-18
          >>>>>>
          >>>>>>> 04:00:45,887
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943 in 0sec
          >>>>>> 2008-07-18 04:00:45,887 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,12160046718422008-07-18
          >>>>>>
          >>>>>>> 04:00:45,892
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842in0sec
          >>>>>> 2008-07-18 04:00:45,892 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Lysander/lang2,12159221274382008-07-18
          >>>>>>
          >>>>>>> 04:00:45,893
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438 in 0sec
          >>>>>> 2008-07-18 04:00:45,893 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791
          >>>>>> 2008-07-18<
          >>>>>> http://sws.geonames.org/5197685/nearby.rdf,12158242597912008-07-18
          >>>>>>
          >>>>>>> 04:00:45,895
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791 in 0sec
          >>>>>> 2008-07-18 04:00:45,895 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://sws.geonames.org/1081116/,1215823957458
          >>>>>> 2008-07-18 <http://sws.geonames.org/1081116/,12158239574582008-07-18
          >>>>>>
          >>>>>>> 04:00:45,895
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://sws.geonames.org/1081116/,1215823957458 in 0sec
          >>>>>> 2008-07-18 04:00:45,896 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/Allomeric,1216039960663
          >>>>>> 2008-07-18 <
          >>>>>> http://dbpedia.org/resource/Allomeric,12160399606632008-07-18
          >>>>>>
          >>>>>>> 04:00:45,935
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/Allomeric,1216039960663 in 0sec
          >>>>>> 2008-07-18 04:00:45,935 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,12157837962592008-07-18
          >>>>>>
          >>>>>>> 04:00:45,993
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259in0sec
          >>>>>> 2008-07-18 04:00:45,994 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,12158238973322008-07-18
          >>>>>>
          >>>>>>> 04:00:46,004
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332 in
          >>>>>> 0sec
          >>>>>> 2008-07-18 04:00:46,005 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,12158242551352008-07-18
          >>>>>>
          >>>>>>> 04:00:46,005
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>> completed on region page-repository,
          >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135 in
          >>>>>> 0sec
          >>>>>> 2008-07-18 04:00:46,005 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> starting compaction on region page-repository,
          >>>>>> http://purl.uniprot.org/taxonomy/112557,1214582151856
          >>>>>> 2008-07-18<
          >>>>>> http://purl.uniprot.org/taxonomy/112557,12145821518562008-07-18
          >>>>>>
          >>>>>>> 04:00:46,007
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
          >>>>>>
          >>>>>> completed on region page-repository,
          >>>>>> http://purl.uniprot.org/taxonomy/112557,1214582151856 in 0sec
          >>>>>> 2008-07-18 04:02:16,798 FATAL
          >>>>>> org.apache.hadoop.hbase.regionserver.Flusher:
          >>>>>> Replay of hlog required. Forcing server restart
          >>>>>> org.apache.hadoop.hbase.DroppedSnapshotException
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1040)
          >>>>>> at
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:942)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:93)
          >>>>>> Caused by: org.apache.hadoop.ipc.RemoteException:
          >>>>>> org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create
          >>>>>> file
          >>>>>>
          >>>>>>
          >>>>>> /hbase/page-repository/1103034897/field/mapfiles/2990615865592052391/index
          >>>>>> for DFSClient_716087201 on client 140.203.154.176 because current
          >>>>>> leaseholder is trying to recreate file.
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:1010)
          >>>>>> at
          >>>>>> org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:967)
          >>>>>> at org.apache.hadoop.dfs.NameNode.create(NameNode.java:269)
          >>>>>> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>>
          >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:557)
          >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
          >>>>>> at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source)
          >>>>>> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
          >>>>>> at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2189)
          >>>>>> at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:479)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.dfs.DistributedFileSystem.create(DistributedFileSystem.java:138)
          >>>>>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:508)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1208)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:402)
          >>>>>> at
          >>>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:307)
          >>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:159)
          >>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:133)
          >>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:125)
          >>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:110)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseWriter.<init>(HStoreFile.java:706)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Writer.<init>(HStoreFile.java:810)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getWriter(HStoreFile.java:453)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:706)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:682)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1030)
          >>>>>> ... 3 more
          >>>>>> 2008-07-18 04:02:16,798 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.Flusher:
          >>>>>> regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
          >>>>>> 2008-07-18 04:02:16,807 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 1 on 60020, call batchUpdate([B@167e3c87, row =>
          >>>>>> http://dbpedia.org/resource/Jurrette_Sindelar,

          {column => field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'}

          ) from 140.203.154.174:50719: error:
          >>>>>> java.io.IOException: Server not running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
          >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:16,847 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 9 on 60020, call batchUpdate([B@525cfe10, row =>
          >>>>>> http://dbpedia.org/resource/Josef_Cyril_Sychra,

          {column => >>>>>> field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'}

          ) from 140.203.154.175:48121: error:
          >>>>>> java.io.IOException: Server not running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
          >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:16,848 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 0 on 60020, call batchUpdate([B@572bf6ae, row =>
          >>>>>> http://dbpedia.org/resource/John_Johnston_%28Royal_Household%29,
          >>>>>>

          {column >>>>>> => field:class, value => '...', column => field:data-source, value => >>>>>> '...', >>>>>> column => field:domain, value => '...', column => >>>>>> field:explicit-content, >>>>>> value => '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'}

          ) from 140.203.154.175:48348: error:
          >>>>>> java.io.IOException: Server not running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
          >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:16,905 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 4 on 60020, call batchUpdate([B@27320a48, row =>
          >>>>>> http://dbpedia.org/resource/Karim_Benkhirane,

          {column => field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'}

          ) from 140.203.154.175:48330: error:
          >>>>>> java.io.IOException: Server not running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
          >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:16,926 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 4 on 60020, call batchUpdate([B@59c7d8c6, row =>
          >>>>>> http://dbpedia.org/resource/Fort_Pierce_%28FL%29,

          {column => >>>>>> field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'}

          ) from 140.203.154.176:58334: error:
          >>>>>> java.io.IOException: Server not running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
          >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:16,959 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 8 on 60020, call batchUpdate([B@4f68dd36, row =>
          >>>>>> http://dbpedia.org/resource/Hendrik_Speuy,

          {column => field:class, >>>>>> value >>>>>> => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'}

          ) from 140.203.154.175:48357: error:
          >>>>>> java.io.IOException: Server not running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
          >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:16,979 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 3 on 60020, call batchUpdate([B@596c7f5f, row =>
          >>>>>> http://dbpedia.org/resource/Harry_Stuck_Sullivan,

          {column => >>>>>> field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'}

          ) from 140.203.154.176:58163: error:
          >>>>>> java.io.IOException: Server not running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
          >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:17,147 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 2 on 60020, call batchUpdate([B@23067cee, row =>
          >>>>>> http://dbpedia.org/resource/High_Littleton,

          {column => field:class, >>>>>> value >>>>>> => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'}

          ) from 140.203.154.174:50952: error:
          >>>>>> java.io.IOException: Server not running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146)
          >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:17,180 INFO org.apache.hadoop.hbase.regionserver.HLog:
          >>>>>> New
          >>>>>> log writer created at
          >>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216350137145
          >>>>>> 2008-07-18 04:02:17,181 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
          >>>>>> 2008-07-18 04:02:17,376 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 7 on 60020, call getClosestRowBefore([B@20d09fed, [B@60b3916f)
          >>>>>> from 140.203.154.175:48464: error: java.io.IOException: Server not
          >>>>>> running
          >>>>>> java.io.IOException: Server not running
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1092)
          >>>>>> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
          >>>>>> at
          >>>>>>
          >>>>>>
          >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
          >>>>>> at
          >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438)
          >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>>>>> 2008-07-18 04:02:17,507 INFO org.apache.hadoop.ipc.Server: Stopping
          >>>>>> server
          >>>>>> on 60020
          >>>>>> 2008-07-18 04:02:17,508 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 1 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,508 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 5 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,587 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 0 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,600 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 2 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,600 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 9 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 8 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 7 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 6 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 4 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server
          >>>>>> handler 3 on 60020: exiting
          >>>>>> 2008-07-18 04:02:17,607 INFO org.apache.hadoop.ipc.Server: Stopping IPC
          >>>>>> Server listener on 60020
          >>>>>> 2008-07-18 04:02:17,691 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
          >>>>>> 2008-07-18 04:02:17,784 INFO org.apache.hadoop.ipc.Server: Stopping IPC
          >>>>>> Server Responder
          >>>>>> 2008-07-18 04:02:17,902 INFO org.mortbay.util.ThreadedServer: Stopping
          >>>>>> Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
          >>>>>> 2008-07-18 04:02:18,179 INFO org.mortbay.http.SocketListener: Stopped
          >>>>>> SocketListener on 0.0.0.0:60030
          >>>>>> 2008-07-18 04:02:28,883 INFO org.mortbay.util.Container: Stopped
          >>>>>> HttpContext[/static,/static]
          >>>>>> 2008-07-18 04:02:31,116 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread
          >>>>>> exiting
          >>>>>> 2008-07-18 04:02:31,117 INFO org.mortbay.util.Container: Stopped
          >>>>>> HttpContext[/logs,/logs]
          >>>>>> 2008-07-18 04:02:31,117 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
          >>>>>> regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting
          >>>>>> 2008-07-18 04:02:31,125 INFO org.mortbay.util.Container: Stopped
          >>>>>> org.mortbay.jetty.servlet.WebApplicationHandler@53797795
          >>>>>> 2008-07-18 04:02:34,017 INFO org.mortbay.util.Container: Stopped
          >>>>>> WebApplicationContext[/,/]
          >>>>>> 2008-07-18 04:02:34,017 INFO org.mortbay.util.Container: Stopped
          >>>>>> org.mortbay.jetty.Server@b8c3679
          >>>>>> 2008-07-18 04:02:34,309 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed
          >>>>>> hlog
          >>>>>> 2008-07-18 04:55:59,098 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> -3929282808018198437 lease expired
          >>>>>> 2008-07-18 04:56:00,258 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> 8694039619106515508 lease expired
          >>>>>> 2008-07-18 04:56:02,068 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> -6642383936144787497 lease expired
          >>>>>> 2008-07-18 04:56:02,178 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> -8674274402855984090 lease expired
          >>>>>> 2008-07-18 04:56:02,858 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> -8684213796880407868 lease expired
          >>>>>> 2008-07-18 04:56:03,478 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> -8003294377021166340 lease expired
          >>>>>> 2008-07-18 04:56:05,298 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> 4741746108028229290 lease expired
          >>>>>> 2008-07-18 04:56:05,418 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> -3662383214249882020 lease expired
          >>>>>> 2008-07-18 04:56:05,498 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
          >>>>>> -5749397523507975233 lease expired
          >>>>>> 2008-07-18 04:56:05,499 INFO org.apache.hadoop.hbase.Leases:
          >>>>>> regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closing leases
          >>>>>> 2008-07-18 04:56:05,499 INFO org.apache.hadoop.hbase.Leases:
          >>>>>> regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closed leases
          >>>>>> 2008-07-18 04:56:05,499 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> closed .META.,,1
          >>>>>> 2008-07-18 04:56:05,499 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
          >>>>>> closed page-repository,
          >>>>>> http://dbpedia.org/resource/CAMBIA,1215970756532
          >>>>>> 2008-07-18 <http://dbpedia.org/resource/CAMBIA,12159707565322008-07-18
          >>>>>>
          >>>>>>> 04:56:05,499
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,12160445949432008-07-18
          >>>>>>
          >>>>>>> 04:56:05,499
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://users.livejournal.com/_alicesprings/data/foaf,1215824861383
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://users.livejournal.com/_alicesprings/data/foaf,12158248613832008-07-18
          >>>>>>
          >>>>>>> 04:56:05,499
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Monrad_Charles_Wallgren,1216118586621
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Monrad_Charles_Wallgren,12161185866212008-07-18
          >>>>>>
          >>>>>>> 04:56:05,500
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,12157837962592008-07-18
          >>>>>>
          >>>>>>> 04:56:05,500
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://sws.geonames.org/1081116/,1215823957458
          >>>>>> 2008-07-18 <http://sws.geonames.org/1081116/,12158239574582008-07-18
          >>>>>>
          >>>>>>> 04:56:05,500
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330
          >>>>>> 2008-07-18<
          >>>>>> http://acm.rkbexplorer.com/id/person-592138,12158248783302008-07-18
          >>>>>>
          >>>>>>> 04:56:05,500
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Microrregi%C3%A3o_de_Ribeir%C3%A3o_Preto,1216118586621
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Microrregi%C3%A3o_de_Ribeir%C3%A3o_Preto,12161185866212008-07-18
          >>>>>>
          >>>>>>> 04:56:05,500
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Jens-Paul_Wollenberg,1216109320961
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Jens-Paul_Wollenberg,12161093209612008-07-18
          >>>>>>
          >>>>>>> 04:56:05,500
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://purl.uniprot.org/go/0009140,1214582602882
          >>>>>> 2008-07-18 <http://purl.uniprot.org/go/0009140,12145826028822008-07-18
          >>>>>>
          >>>>>>> 04:56:05,500
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,12145475938142008-07-18
          >>>>>>
          >>>>>>> 04:56:05,500
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://bio2rdf.org/omim:300357,1215825157318
          >>>>>> 2008-07-18 <http://bio2rdf.org/omim:300357,12158251573182008-07-18
          >>>>>>
          >>>>>>> 04:56:05,501
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://alcidesfonseca.com/blog/35,1215825925595
          >>>>>> 2008-07-18 <http://alcidesfonseca.com/blog/35,12158259255952008-07-18
          >>>>>>
          >>>>>>> 04:56:05,501
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://dbpedia.org/resource/Allomeric,1216039960663
          >>>>>> 2008-07-18 <
          >>>>>> http://dbpedia.org/resource/Allomeric,12160399606632008-07-18
          >>>>>>
          >>>>>>> 04:56:05,501
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791
          >>>>>> 2008-07-18<
          >>>>>> http://sws.geonames.org/5197685/nearby.rdf,12158242597912008-07-18
          >>>>>>
          >>>>>>> 04:56:05,501
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Portland__Garden_Club,1216125563046
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Portland__Garden_Club,12161255630462008-07-18
          >>>>>>
          >>>>>>> 04:56:05,501
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/John_J._Bernet,1216109320961
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/John_J._Bernet,12161093209612008-07-18
          >>>>>>
          >>>>>>> 04:56:05,501
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Flowers_of_Life,12160084595652008-07-18
          >>>>>>
          >>>>>>> 04:56:05,501
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://sws.geonames.org/3025251/,1215824259058
          >>>>>> 2008-07-18 <http://sws.geonames.org/3025251/,12158242590582008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,12161255630462008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Billy_Forester,1216165245424
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Billy_Forester,12161652454242008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,12159192644012008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://dbpedia.org/resource/Snatcho,1216263402275
          >>>>>> 2008-07-18 <
          >>>>>> http://dbpedia.org/resource/Snatcho,12162634022752008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,12158242551352008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Template:superheroe,1216140016580
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Template:superheroe,12161400165802008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,12158238973322008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,12158238964112008-07-18
          >>>>>>
          >>>>>>> 04:56:05,502
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,12145963998112008-07-18
          >>>>>>
          >>>>>>> 04:56:05,503
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://dbpedia.org/resource/Bob_Errey,1216165245424
          >>>>>> 2008-07-18 <
          >>>>>> http://dbpedia.org/resource/Bob_Errey,12161652454242008-07-18
          >>>>>>
          >>>>>>> 04:56:05,503
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/The_Way_To_Fight,1216138073477
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/The_Way_To_Fight,12161380734772008-07-18
          >>>>>>
          >>>>>>> 04:56:05,503
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Al_Fateh-universitetet,1215959690520
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Al_Fateh-universitetet,12159596905202008-07-18
          >>>>>>
          >>>>>>> 04:56:05,503
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://dbpedia.org/resource/Syed_Koteb,1216140016580
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Syed_Koteb,12161400165802008-07-18
          >>>>>>
          >>>>>>> 04:56:05,503
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,12159157700092008-07-18
          >>>>>>
          >>>>>>> 04:56:05,503
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://dbpedia.org/resource/1er_RCP,1215980565468
          >>>>>> 2008-07-18 <
          >>>>>> http://dbpedia.org/resource/1er_RCP,12159805654682008-07-18
          >>>>>>
          >>>>>>> 04:56:05,503
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://purl.uniprot.org/taxonomy/112557,1214582151856
          >>>>>> 2008-07-18<
          >>>>>> http://purl.uniprot.org/taxonomy/112557,12145821518562008-07-18
          >>>>>>
          >>>>>>> 04:56:05,504
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Harry_Girvetz,1215910087698
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Harry_Girvetz,12159100876982008-07-18
          >>>>>>
          >>>>>>> 04:56:05,504
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Liste_der_Autobahnen_in_Spanien,1215922127438
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Liste_der_Autobahnen_in_Spanien,12159221274382008-07-18
          >>>>>>
          >>>>>>> 04:56:05,504
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://apassant.net/home/2007/12/flickrdf/data/groups/355509@N23,1215825925595
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://apassant.net/home/2007/12/flickrdf/data/groups/355509@N23,12158259255952008-07-18
          >>>>>>
          >>>>>>> 04:56:05,504
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Category:1948_in_law,1215896373168
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Category:1948_in_law,12158963731682008-07-18
          >>>>>>
          >>>>>>> 04:56:05,504
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://acm.rkbexplorer.com/id/person-94180,1215825860040
          >>>>>> 2008-07-18<
          >>>>>> http://acm.rkbexplorer.com/id/person-94180,12158258600402008-07-18
          >>>>>>
          >>>>>>> 04:56:05,504
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://www.akropolix.net/rik0/blogs/category/mac/,1215823941299
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://www.akropolix.net/rik0/blogs/category/mac/,12158239412992008-07-18
          >>>>>>
          >>>>>>> 04:56:05,504
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Trevor_Ashmore,1216138073477
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Trevor_Ashmore,12161380734772008-07-18
          >>>>>>
          >>>>>>> 04:56:05,504
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Papa_New_Guinea_Baseball_Federation,1215929139300
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Papa_New_Guinea_Baseball_Federation,12159291393002008-07-18
          >>>>>>
          >>>>>>> 04:56:05,505
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,12159472244252008-07-18
          >>>>>>
          >>>>>>> 04:56:05,505
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,12160399606632008-07-18
          >>>>>>
          >>>>>>> 04:56:05,505
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://dbpedia.org/resource/1960-09-03,1215980565468
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/1960-09-03,12159805654682008-07-18
          >>>>>>
          >>>>>>> 04:56:05,505
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,12159589559112008-07-18
          >>>>>>
          >>>>>>> 04:56:05,505
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,12159830915392008-07-18
          >>>>>>
          >>>>>>> 04:56:05,505
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>>
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842
          >>>>>> 2008-07-18<
          >>>>>>
          >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,12160046718422008-07-18
          >>>>>>
          >>>>>>> 04:56:05,505
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Lysander/lang2,12159221274382008-07-18
          >>>>>>
          >>>>>>> 04:56:05,506
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://sws.geonames.org/2701307/,1215823954578
          >>>>>> 2008-07-18 <http://sws.geonames.org/2701307/,12158239545782008-07-18
          >>>>>>
          >>>>>>> 04:56:05,506
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,
          >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,12158239811372008-07-18
          >>>>>>
          >>>>>>> 04:56:05,506
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed
          >>>>>> page-repository,http://dbpedia.org/resource/Angus_McKay,1215890852482
          >>>>>> 2008-07-18<
          >>>>>> http://dbpedia.org/resource/Angus_McKay,12158908524822008-07-18
          >>>>>>
          >>>>>>> 04:56:05,506
          >>>>>>>
          >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
          >>>>>> aborting server at: 140.203.154.176:60020
          >>>>>> 2008-07-18 04:56:05,506 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
          >>>>>> regionserver/0:0:0:0:0:0:0:0:60020 exiting
          >>>>>> 2008-07-18 04:56:05,561 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
          >>>>>> thread.
          >>>>>> 2008-07-18 04:56:05,562 INFO
          >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread
          >>>>>> complete
          >>>>>>
          >>>>>>
          >>>>>>
          >>>>>>
          >>>>>>
          >>>>>>
          >>>>>>
          >>>>>>
          >>>>>
          >>>>>
          >>>
          >> 2008-07-19 07:31:45,505 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
          >> Exception: java.net.SocketTimeoutException: timed out waiting for rpc
          >> response
          >> 2008-07-19 07:31:45,506 WARN org.apache.hadoop.dfs.DFSClient: Error
          >> Recovery for block null bad datanode[0]
          >> 2008-07-19 07:31:54,868 FATAL org.apache.hadoop.hbase.regionserver.Flusher:
          >> Replay of hlog required. Forcing server restart
          >> org.apache.hadoop.hbase.DroppedSnapshotException
          >> at
          >> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1040)
          >> at
          >> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:942)
          >> at
          >> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174)
          >> at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:93)
          >> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
          >> Could not complete write to file
          >> /hbase/page-repository/963177038/field/info/4591605661998875281 by
          >> DFSClient_1009756630
          >> at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:332)
          >> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
          >> at
          >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >> at java.lang.reflect.Method.invoke(Method.java:597)
          >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
          >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
          >>
          >> at org.apache.hadoop.ipc.Client.call(Client.java:557)
          >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
          >> at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source)
          >> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
          >> at
          >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          >> at java.lang.reflect.Method.invoke(Method.java:597)
          >> at
          >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
          >> at
          >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
          >> at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source)
          >> at
          >> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2655)
          >> at
          >> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2576)
          >> at
          >> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
          >> at
          >> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
          >> at
          >> org.apache.hadoop.hbase.regionserver.HStoreFile.writeInfo(HStoreFile.java:350)
          >> at
          >> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:747)
          >> at
          >> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:682)
          >> at
          >> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1030)
          >> ... 3 more
          >> 2008-07-19 07:31:54,868 INFO org.apache.hadoop.hbase.regionserver.Flusher:
          >> regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
          >>
          >>
          >>
          >
          >

          Show
          Andrew Purtell added a comment - From hbase-users@ From: "Renaud Delbru" <renaud.delbru@deri.org> To: hbase-user@hadoop.apache.org Saturday, July 19, 2008 7:46 AM The cluster is currently composed of 3 nodes. The cluster is also used for executing hadoop jobs. One of the node acts as an hadoop master, an HMaster and an HRegionServer. The current settings are: Xmx=1024M for the Hadoop Master, Xmx=2048M for the HMaster, Xmx=2048M for each HRegionServer. The inserts are somehow randomised. We have 24 hadoop jobs uploading various documents. The row key is the URL of the document, and each hadoop task has to upload a range of URLs (for example, http://my-url/AAA - http://my-url/BBB ). Thus, I assume that each hadoop tasks will hit a different region and that the insert load is distributed among the nodes. Thanks. – Renaud Delbru Jean-Daniel Cryans wrote: > Ok then, at least having the nodes synchronized keeps you safe from other > problems. By the way, how many nodes do you have? > > One other thing to check is your writing pattern. If you insert data with > sequential row keys, an example would be designing your row keys as an > incremented primary key equivalent but left padded with zeros, these inserts > will always hit on a single region thus the load will not be distributed. If > it's your case, try randomizing the inserts. > > I will also look at your logs monday morning. > > Thx, > > J-D > > On Sat, Jul 19, 2008 at 5:47 AM, Renaud Delbru <renaud.delbru@deri.org> > wrote: > > >> Hi Jean-Daniel, >> >> I have synchronised all the nodes to the same ntp server yesterday. >> This morning, another regionserver went down for the same reason (log >> attached). >> >> Thanks. >> – >> Renaud Delbru >> >> >> >> Jean-Daniel Cryans wrote: >> >> >>> Renaud, >>> >>> Yes, the impact is important. Try fixing it first because typically heavy >>> write load problems only occur when there is a very few number of nodes >>> and >>> regions. We have seen this many times. >>> >>> Happy fixing! >>> >>> J-D >>> >>> On Fri, Jul 18, 2008 at 5:20 PM, Renaud Delbru <renaud.delbru@deri.org> >>> wrote: >>> >>> >>> >>> >>>> Hi Jean-Daniel, >>>> >>>> my nodes clocks are desynchronised (each node has a different time). >>>> Could >>>> it have an impact on HBase ? >>>> >>>> At the moment of the failure, the node had about 55 regions (with >>>> hbase.hregion.max.filesize = 128M). >>>> >>>> Thanks. >>>> – >>>> Renaud Delbru >>>> >>>> As One node is at >>>> >>>> Jean-Daniel Cryans wrote: >>>> >>>> >>>> >>>> >>>>> Renaud, >>>>> >>>>> Just to make sure, are all your nodes clocks synchronized? Also, at the >>>>> moment of that failure, how many regions did the table have? >>>>> >>>>> Thanks, >>>>> >>>>> J-D >>>>> >>>>> On Fri, Jul 18, 2008 at 5:11 AM, Renaud Delbru <renaud.delbru@deri.org> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Hi, >>>>>> >>>>>> I am using HBase 0.2.0-dev, Hudson Build #208. >>>>>> I am experiencing a problem with HBase that looks like the issue >>>>>> https://issues.apache.org/jira/browse/HBASE-15 . >>>>>> During an upload intensive task, one of the regionserver went down to >>>>>> force rerun of hlog. >>>>>> >>>>>> This is not the first time we have experienced this problem, it >>>>>> happened >>>>>> already on another cluster node a few days ago, in the same settings. >>>>>> >>>>>> I have attached the regionserver log. I don't notice something >>>>>> particular in the hadoop datanode log. >>>>>> >>>>>> Thanks in advance for any advices. >>>>>> – >>>>>> Renaud Delbru >>>>>> >>>>>> >>>>>> >>>>>> 2008-07-18 03:40:51,861 INFO >>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of >>>>>> entries: 30008 >>>>>> 2008-07-18 03:40:52,073 INFO org.apache.hadoop.hbase.regionserver.HLog: >>>>>> New >>>>>> log writer created at >>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216348851959 >>>>>> 2008-07-18 03:43:36,257 INFO >>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of >>>>>> entries: 30008 >>>>>> 2008-07-18 03:43:37,020 INFO org.apache.hadoop.hbase.regionserver.HLog: >>>>>> New >>>>>> log writer created at >>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216349016755 >>>>>> 2008-07-18 03:45:12,963 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,12159192644012008-07-18 >>>>>> >>>>>>> 03:45:58,868 >>>>>>> >>>>>> WARN org.apache.hadoop.hbase.util.Sleeper: We slept 30837ms, >>>>>> >>>>>> ten times longer than scheduled: 3000 >>>>>> 2008-07-18 03:55:57,783 WARN org.apache.hadoop.hbase.util.Sleeper: We >>>>>> slept >>>>>> 583701ms, ten times longer than scheduled: 3000 >>>>>> 2008-07-18 03:55:57,787 INFO >>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: Rolling hlog. Number of >>>>>> entries: 21011 >>>>>> 2008-07-18 03:55:58,410 WARN org.apache.hadoop.dfs.DFSClient: Exception >>>>>> while reading from blk_-6424749704096696952 of >>>>>> >>>>>> /hbase/page-repository/942413588/field/mapfiles/7570604533190594376/data >>>>>> from 140.203.154.176:50010: java.io.IOException: Premeture EOF from >>>>>> inputStream >>>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) >>>>>> at >>>>>> >>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236) >>>>>> at >>>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191) >>>>>> at >>>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159) >>>>>> at >>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:829) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1352) >>>>>> at >>>>>> >>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1388) >>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:178) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56) >>>>>> at >>>>>> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1643) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBlock(SequenceFile.java:1682) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1799) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1828) >>>>>> at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:499) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.MapFileCompactionReader.next(MapFileCompactionReader.java:45) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStore.compactHStoreFiles(HStore.java:1029) >>>>>> at >>>>>> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:896) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:880) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:840) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:84) >>>>>> >>>>>> 2008-07-18 03:55:58,966 WARN org.apache.hadoop.dfs.DFSClient: Exception >>>>>> while reading from blk_7305122468419183134 of >>>>>> >>>>>> /hbase/page-repository/942413588/field/mapfiles/4454700123966493614/data >>>>>> from 140.203.154.176:50010: java.io.IOException: Premeture EOF from >>>>>> inputStream >>>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) >>>>>> at >>>>>> >>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236) >>>>>> at >>>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191) >>>>>> at >>>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159) >>>>>> at >>>>>> org.apache.hadoop.dfs.DFSClient$BlockReader.read(DFSClient.java:829) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1352) >>>>>> at >>>>>> >>>>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1388) >>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:178) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56) >>>>>> at >>>>>> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1643) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1709) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1745) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1831) >>>>>> at org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:499) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.MapFileCompactionReader.next(MapFileCompactionReader.java:45) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStore.compactHStoreFiles(HStore.java:1029) >>>>>> at >>>>>> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:896) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:880) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:840) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:84) >>>>>> >>>>>> 2008-07-18 03:56:05,305 INFO org.apache.hadoop.hbase.regionserver.HLog: >>>>>> New >>>>>> log writer created at >>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216349764848 >>>>>> 2008-07-18 03:57:30,244 ERROR >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening >>>>>> scanner >>>>>> (fsOk: true) >>>>>> java.io.IOException: HStoreScanner failed construction >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:69) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:68) >>>>>> at >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1896) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1767) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1293) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1186) >>>>>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc >>>>>> response >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:559) >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) >>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source) >>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source) >>>>>> at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:572) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:365) >>>>>> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:628) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.createDataFileReader(HStoreFile.java:677) >>>>>> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:659) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:644) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:734) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:409) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:95) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:66) >>>>>> ... 10 more >>>>>> 2008-07-18 03:58:01,895 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 7 on 60020, call openScanner([B@9dd0d92, [[B@fa5f14d, >>>>>> [B@5784fe41, >>>>>> 9223372036854775807, null) from 140.203.154.174:48958: error: >>>>>> java.io.IOException: HStoreScanner failed construction >>>>>> java.io.IOException: HStoreScanner failed construction >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:69) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:68) >>>>>> at >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1896) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1767) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1293) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1186) >>>>>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc >>>>>> response >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:559) >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) >>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source) >>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >>>>>> at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source) >>>>>> at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:572) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:365) >>>>>> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:628) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.createDataFileReader(HStoreFile.java:677) >>>>>> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:659) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:644) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:734) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:409) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:95) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:66) >>>>>> ... 10 more >>>>>> 2008-07-18 03:58:06,728 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> compaction completed on region page-repository, >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401 in 12mins, >>>>>> 53sec >>>>>> 2008-07-18 03:58:06,728 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Angus_McKay,1215890852482 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Angus_McKay,12158908524822008-07-18 >>>>>> >>>>>>> 03:58:06,729 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/Angus_McKay,1215890852482 in 0sec >>>>>> 2008-07-18 03:58:06,729 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,12159472244252008-07-18 >>>>>> >>>>>>> 03:58:06,730 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425 in 0sec >>>>>> 2008-07-18 03:58:06,730 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,12159830915392008-07-18 >>>>>> >>>>>>> 03:58:06,760 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539in0sec >>>>>> 2008-07-18 03:58:06,760 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> >>>>>> >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811 >>>>>> 2008-07-18< >>>>>> >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,12145963998112008-07-18 >>>>>> >>>>>>> 03:58:06,760 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> >>>>>> >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811in0sec >>>>>> 2008-07-18 03:58:06,761 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,12159589559112008-07-18 >>>>>> >>>>>>> 03:58:06,761 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911 in 0sec >>>>>> 2008-07-18 03:58:06,761 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,12158238964112008-07-18 >>>>>> >>>>>>> 03:58:06,777 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411in0sec >>>>>> 2008-07-18 03:58:06,777 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,12158239811372008-07-18 >>>>>> >>>>>>> 03:58:06,800 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137 in 0sec >>>>>> 2008-07-18 03:58:06,800 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,12160399606632008-07-18 >>>>>> >>>>>>> 03:58:06,826 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663in >>>>>> 0sec >>>>>> 2008-07-18 03:58:06,826 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> >>>>>> >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814 >>>>>> 2008-07-18< >>>>>> >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,12145475938142008-07-18 >>>>>> >>>>>>> 03:58:06,916 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> >>>>>> >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814in0sec >>>>>> 2008-07-18 03:58:06,916 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://sws.geonames.org/3025251/,1215824259058 >>>>>> 2008-07-18 < http://sws.geonames.org/3025251/,12158242590582008-07-18 >>>>>> >>>>>>> 03:58:06,972 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://sws.geonames.org/3025251/,1215824259058 in 0sec >>>>>> 2008-07-18 03:58:06,973 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,12161255630462008-07-18 >>>>>> >>>>>>> 03:58:06,973 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046in >>>>>> 0sec >>>>>> 2008-07-18 03:58:06,973 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Flowers_of_Life,12160084595652008-07-18 >>>>>> >>>>>>> 03:58:15,979 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling >>>>>> hlog. Number of entries: 30009 >>>>>> 2008-07-18 03:59:32,093 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> compaction completed on region page-repository, >>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565 in 1mins, >>>>>> 25sec >>>>>> 2008-07-18 03:59:32,093 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330 >>>>>> 2008-07-18< >>>>>> http://acm.rkbexplorer.com/id/person-592138,12158248783302008-07-18 >>>>>> >>>>>>> 03:59:32,134 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330 in 0sec >>>>>> 2008-07-18 03:59:32,134 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,12159157700092008-07-18 >>>>>> >>>>>>> 04:00:45,884 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009 in >>>>>> 1mins, >>>>>> 13sec >>>>>> 2008-07-18 04:00:45,885 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,12160445949432008-07-18 >>>>>> >>>>>>> 04:00:45,887 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943 in 0sec >>>>>> 2008-07-18 04:00:45,887 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> >>>>>> >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,12160046718422008-07-18 >>>>>> >>>>>>> 04:00:45,892 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> >>>>>> >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842in0sec >>>>>> 2008-07-18 04:00:45,892 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Lysander/lang2,12159221274382008-07-18 >>>>>> >>>>>>> 04:00:45,893 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438 in 0sec >>>>>> 2008-07-18 04:00:45,893 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791 >>>>>> 2008-07-18< >>>>>> http://sws.geonames.org/5197685/nearby.rdf,12158242597912008-07-18 >>>>>> >>>>>>> 04:00:45,895 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791 in 0sec >>>>>> 2008-07-18 04:00:45,895 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://sws.geonames.org/1081116/,1215823957458 >>>>>> 2008-07-18 < http://sws.geonames.org/1081116/,12158239574582008-07-18 >>>>>> >>>>>>> 04:00:45,895 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://sws.geonames.org/1081116/,1215823957458 in 0sec >>>>>> 2008-07-18 04:00:45,896 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/Allomeric,1216039960663 >>>>>> 2008-07-18 < >>>>>> http://dbpedia.org/resource/Allomeric,12160399606632008-07-18 >>>>>> >>>>>>> 04:00:45,935 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/Allomeric,1216039960663 in 0sec >>>>>> 2008-07-18 04:00:45,935 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> >>>>>> >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,12157837962592008-07-18 >>>>>> >>>>>>> 04:00:45,993 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> >>>>>> >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259in0sec >>>>>> 2008-07-18 04:00:45,994 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,12158238973322008-07-18 >>>>>> >>>>>>> 04:00:46,004 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332 in >>>>>> 0sec >>>>>> 2008-07-18 04:00:46,005 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,12158242551352008-07-18 >>>>>> >>>>>>> 04:00:46,005 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> completed on region page-repository, >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135 in >>>>>> 0sec >>>>>> 2008-07-18 04:00:46,005 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> starting compaction on region page-repository, >>>>>> http://purl.uniprot.org/taxonomy/112557,1214582151856 >>>>>> 2008-07-18< >>>>>> http://purl.uniprot.org/taxonomy/112557,12145821518562008-07-18 >>>>>> >>>>>>> 04:00:46,007 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction >>>>>> >>>>>> completed on region page-repository, >>>>>> http://purl.uniprot.org/taxonomy/112557,1214582151856 in 0sec >>>>>> 2008-07-18 04:02:16,798 FATAL >>>>>> org.apache.hadoop.hbase.regionserver.Flusher: >>>>>> Replay of hlog required. Forcing server restart >>>>>> org.apache.hadoop.hbase.DroppedSnapshotException >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1040) >>>>>> at >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:942) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174) >>>>>> at >>>>>> org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:93) >>>>>> Caused by: org.apache.hadoop.ipc.RemoteException: >>>>>> org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create >>>>>> file >>>>>> >>>>>> >>>>>> /hbase/page-repository/1103034897/field/mapfiles/2990615865592052391/index >>>>>> for DFSClient_716087201 on client 140.203.154.176 because current >>>>>> leaseholder is trying to recreate file. >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:1010) >>>>>> at >>>>>> org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:967) >>>>>> at org.apache.hadoop.dfs.NameNode.create(NameNode.java:269) >>>>>> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:557) >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) >>>>>> at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source) >>>>>> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >>>>>> at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2189) >>>>>> at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:479) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.dfs.DistributedFileSystem.create(DistributedFileSystem.java:138) >>>>>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:508) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1208) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:402) >>>>>> at >>>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:307) >>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:159) >>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:133) >>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:125) >>>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:110) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseWriter.<init>(HStoreFile.java:706) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Writer.<init>(HStoreFile.java:810) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStoreFile.getWriter(HStoreFile.java:453) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:706) >>>>>> at >>>>>> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:682) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1030) >>>>>> ... 3 more >>>>>> 2008-07-18 04:02:16,798 INFO >>>>>> org.apache.hadoop.hbase.regionserver.Flusher: >>>>>> regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting >>>>>> 2008-07-18 04:02:16,807 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 1 on 60020, call batchUpdate([B@167e3c87, row => >>>>>> http://dbpedia.org/resource/Jurrette_Sindelar , {column => field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'} ) from 140.203.154.174:50719: error: >>>>>> java.io.IOException: Server not running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146) >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:16,847 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 9 on 60020, call batchUpdate([B@525cfe10, row => >>>>>> http://dbpedia.org/resource/Josef_Cyril_Sychra , {column => >>>>>> field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'} ) from 140.203.154.175:48121: error: >>>>>> java.io.IOException: Server not running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146) >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:16,848 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 0 on 60020, call batchUpdate([B@572bf6ae, row => >>>>>> http://dbpedia.org/resource/John_Johnston_%28Royal_Household%29 , >>>>>> {column >>>>>> => field:class, value => '...', column => field:data-source, value => >>>>>> '...', >>>>>> column => field:domain, value => '...', column => >>>>>> field:explicit-content, >>>>>> value => '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'} ) from 140.203.154.175:48348: error: >>>>>> java.io.IOException: Server not running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146) >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:16,905 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 4 on 60020, call batchUpdate([B@27320a48, row => >>>>>> http://dbpedia.org/resource/Karim_Benkhirane , {column => field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'} ) from 140.203.154.175:48330: error: >>>>>> java.io.IOException: Server not running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146) >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:16,926 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 4 on 60020, call batchUpdate([B@59c7d8c6, row => >>>>>> http://dbpedia.org/resource/Fort_Pierce_%28FL%29 , {column => >>>>>> field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'} ) from 140.203.154.176:58334: error: >>>>>> java.io.IOException: Server not running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146) >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:16,959 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 8 on 60020, call batchUpdate([B@4f68dd36, row => >>>>>> http://dbpedia.org/resource/Hendrik_Speuy , {column => field:class, >>>>>> value >>>>>> => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'} ) from 140.203.154.175:48357: error: >>>>>> java.io.IOException: Server not running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146) >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:16,979 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 3 on 60020, call batchUpdate([B@596c7f5f, row => >>>>>> http://dbpedia.org/resource/Harry_Stuck_Sullivan , {column => >>>>>> field:class, >>>>>> value => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'} ) from 140.203.154.176:58163: error: >>>>>> java.io.IOException: Server not running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146) >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:17,147 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 2 on 60020, call batchUpdate([B@23067cee, row => >>>>>> http://dbpedia.org/resource/High_Littleton , {column => field:class, >>>>>> value >>>>>> => '...', column => field:data-source, value => '...', column => >>>>>> field:domain, value => '...', column => field:explicit-content, value >>>>>> => >>>>>> '...', column => field:format, value => '...', column => >>>>>> field:implicit-content, value => '...', column => field:label, value => >>>>>> '...', column => field:length, value => '...', column => >>>>>> field:ontology, >>>>>> value => '...', column => field:size, value => '...', column => >>>>>> field:timestamp, value => '...'} ) from 140.203.154.174:50952: error: >>>>>> java.io.IOException: Server not running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1146) >>>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:17,180 INFO org.apache.hadoop.hbase.regionserver.HLog: >>>>>> New >>>>>> log writer created at >>>>>> /hbase/log_140.203.154.176_1216023782181_60020/hlog.dat.1216350137145 >>>>>> 2008-07-18 04:02:17,181 INFO >>>>>> org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting. >>>>>> 2008-07-18 04:02:17,376 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 7 on 60020, call getClosestRowBefore([B@20d09fed, [B@60b3916f) >>>>>> from 140.203.154.175:48464: error: java.io.IOException: Server not >>>>>> running >>>>>> java.io.IOException: Server not running >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:1447) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1092) >>>>>> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) >>>>>> at >>>>>> >>>>>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:438) >>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>>>> 2008-07-18 04:02:17,507 INFO org.apache.hadoop.ipc.Server: Stopping >>>>>> server >>>>>> on 60020 >>>>>> 2008-07-18 04:02:17,508 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 1 on 60020: exiting >>>>>> 2008-07-18 04:02:17,508 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 5 on 60020: exiting >>>>>> 2008-07-18 04:02:17,587 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 0 on 60020: exiting >>>>>> 2008-07-18 04:02:17,600 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 2 on 60020: exiting >>>>>> 2008-07-18 04:02:17,600 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 9 on 60020: exiting >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 8 on 60020: exiting >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 7 on 60020: exiting >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 6 on 60020: exiting >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 4 on 60020: exiting >>>>>> 2008-07-18 04:02:17,601 INFO org.apache.hadoop.ipc.Server: IPC Server >>>>>> handler 3 on 60020: exiting >>>>>> 2008-07-18 04:02:17,607 INFO org.apache.hadoop.ipc.Server: Stopping IPC >>>>>> Server listener on 60020 >>>>>> 2008-07-18 04:02:17,691 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer >>>>>> 2008-07-18 04:02:17,784 INFO org.apache.hadoop.ipc.Server: Stopping IPC >>>>>> Server Responder >>>>>> 2008-07-18 04:02:17,902 INFO org.mortbay.util.ThreadedServer: Stopping >>>>>> Acceptor ServerSocket [addr=0.0.0.0/0.0.0.0,port=0,localport=60030] >>>>>> 2008-07-18 04:02:18,179 INFO org.mortbay.http.SocketListener: Stopped >>>>>> SocketListener on 0.0.0.0:60030 >>>>>> 2008-07-18 04:02:28,883 INFO org.mortbay.util.Container: Stopped >>>>>> HttpContext [/static,/static] >>>>>> 2008-07-18 04:02:31,116 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread >>>>>> exiting >>>>>> 2008-07-18 04:02:31,117 INFO org.mortbay.util.Container: Stopped >>>>>> HttpContext [/logs,/logs] >>>>>> 2008-07-18 04:02:31,117 INFO >>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: >>>>>> regionserver/0:0:0:0:0:0:0:0:60020.compactor exiting >>>>>> 2008-07-18 04:02:31,125 INFO org.mortbay.util.Container: Stopped >>>>>> org.mortbay.jetty.servlet.WebApplicationHandler@53797795 >>>>>> 2008-07-18 04:02:34,017 INFO org.mortbay.util.Container: Stopped >>>>>> WebApplicationContext [/,/] >>>>>> 2008-07-18 04:02:34,017 INFO org.mortbay.util.Container: Stopped >>>>>> org.mortbay.jetty.Server@b8c3679 >>>>>> 2008-07-18 04:02:34,309 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed >>>>>> hlog >>>>>> 2008-07-18 04:55:59,098 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> -3929282808018198437 lease expired >>>>>> 2008-07-18 04:56:00,258 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> 8694039619106515508 lease expired >>>>>> 2008-07-18 04:56:02,068 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> -6642383936144787497 lease expired >>>>>> 2008-07-18 04:56:02,178 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> -8674274402855984090 lease expired >>>>>> 2008-07-18 04:56:02,858 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> -8684213796880407868 lease expired >>>>>> 2008-07-18 04:56:03,478 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> -8003294377021166340 lease expired >>>>>> 2008-07-18 04:56:05,298 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> 4741746108028229290 lease expired >>>>>> 2008-07-18 04:56:05,418 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> -3662383214249882020 lease expired >>>>>> 2008-07-18 04:56:05,498 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>>> -5749397523507975233 lease expired >>>>>> 2008-07-18 04:56:05,499 INFO org.apache.hadoop.hbase.Leases: >>>>>> regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closing leases >>>>>> 2008-07-18 04:56:05,499 INFO org.apache.hadoop.hbase.Leases: >>>>>> regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closed leases >>>>>> 2008-07-18 04:56:05,499 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> closed .META.,,1 >>>>>> 2008-07-18 04:56:05,499 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>> closed page-repository, >>>>>> http://dbpedia.org/resource/CAMBIA,1215970756532 >>>>>> 2008-07-18 < http://dbpedia.org/resource/CAMBIA,12159707565322008-07-18 >>>>>> >>>>>>> 04:56:05,499 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,1216044594943 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Brachyotum_fictum,12160445949432008-07-18 >>>>>> >>>>>>> 04:56:05,499 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://users.livejournal.com/_alicesprings/data/foaf,1215824861383 >>>>>> 2008-07-18< >>>>>> >>>>>> http://users.livejournal.com/_alicesprings/data/foaf,12158248613832008-07-18 >>>>>> >>>>>>> 04:56:05,499 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Monrad_Charles_Wallgren,1216118586621 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Monrad_Charles_Wallgren,12161185866212008-07-18 >>>>>> >>>>>>> 04:56:05,500 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,1215783796259 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/%E7%8E%96%E7%8F%82%E3%83%91%E3%83%BC%E3%82%AD%E3%83%B3%E3%82%B0%E3%82%A8%E3%83%AA%E3%82%A2,12157837962592008-07-18 >>>>>> >>>>>>> 04:56:05,500 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://sws.geonames.org/1081116/,1215823957458 >>>>>> 2008-07-18 < http://sws.geonames.org/1081116/,12158239574582008-07-18 >>>>>> >>>>>>> 04:56:05,500 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://acm.rkbexplorer.com/id/person-592138,1215824878330 >>>>>> 2008-07-18< >>>>>> http://acm.rkbexplorer.com/id/person-592138,12158248783302008-07-18 >>>>>> >>>>>>> 04:56:05,500 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://dbpedia.org/resource/Microrregi%C3%A3o_de_Ribeir%C3%A3o_Preto,1216118586621 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Microrregi%C3%A3o_de_Ribeir%C3%A3o_Preto,12161185866212008-07-18 >>>>>> >>>>>>> 04:56:05,500 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Jens-Paul_Wollenberg,1216109320961 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Jens-Paul_Wollenberg,12161093209612008-07-18 >>>>>> >>>>>>> 04:56:05,500 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://purl.uniprot.org/go/0009140,1214582602882 >>>>>> 2008-07-18 < http://purl.uniprot.org/go/0009140,12145826028822008-07-18 >>>>>> >>>>>>> 04:56:05,500 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,1214547593814 >>>>>> 2008-07-18< >>>>>> >>>>>> http://riese.joanneum.at/data/dca17168/2001_wrl_rest_fgdp_555_de,12145475938142008-07-18 >>>>>> >>>>>>> 04:56:05,500 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://bio2rdf.org/omim:300357,1215825157318 >>>>>> 2008-07-18 < http://bio2rdf.org/omim:300357,12158251573182008-07-18 >>>>>> >>>>>>> 04:56:05,501 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://alcidesfonseca.com/blog/35,1215825925595 >>>>>> 2008-07-18 < http://alcidesfonseca.com/blog/35,12158259255952008-07-18 >>>>>> >>>>>>> 04:56:05,501 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://dbpedia.org/resource/Allomeric,1216039960663 >>>>>> 2008-07-18 < >>>>>> http://dbpedia.org/resource/Allomeric,12160399606632008-07-18 >>>>>> >>>>>>> 04:56:05,501 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://sws.geonames.org/5197685/nearby.rdf,1215824259791 >>>>>> 2008-07-18< >>>>>> http://sws.geonames.org/5197685/nearby.rdf,12158242597912008-07-18 >>>>>> >>>>>>> 04:56:05,501 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Portland__Garden_Club,1216125563046 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Portland__Garden_Club,12161255630462008-07-18 >>>>>> >>>>>>> 04:56:05,501 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/John_J._Bernet,1216109320961 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/John_J._Bernet,12161093209612008-07-18 >>>>>> >>>>>>> 04:56:05,501 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Flowers_of_Life,1216008459565 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Flowers_of_Life,12160084595652008-07-18 >>>>>> >>>>>>> 04:56:05,501 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://sws.geonames.org/3025251/,1215824259058 >>>>>> 2008-07-18 < http://sws.geonames.org/3025251/,12158242590582008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,1216125563046 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Philippe_Le_Hardy_de_Beaulieu,12161255630462008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Billy_Forester,1216165245424 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Billy_Forester,12161652454242008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,1215919264401 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Judah_ibn_Ghayyat,12159192644012008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://dbpedia.org/resource/Snatcho,1216263402275 >>>>>> 2008-07-18 < >>>>>> http://dbpedia.org/resource/Snatcho,12162634022752008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,1215824255135 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/%2837715%29_1996_RN31,12158242551352008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Template:superheroe,1216140016580 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Template:superheroe,12161400165802008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,1215823897332 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/conf/cocoon/CohenCF01,12158238973322008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,1215823896411 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-42151d70d8c1a0a74307093179b4a1f4-c2bbeaffc138db617c870e19b352ce16,12158238964112008-07-18 >>>>>> >>>>>>> 04:56:05,502 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,1214596399811 >>>>>> 2008-07-18< >>>>>> >>>>>> http://paguntaka.org/2007/08/09/polar-star-mining-corporation-reverse-take-over-transaction/,12145963998112008-07-18 >>>>>> >>>>>>> 04:56:05,503 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://dbpedia.org/resource/Bob_Errey,1216165245424 >>>>>> 2008-07-18 < >>>>>> http://dbpedia.org/resource/Bob_Errey,12161652454242008-07-18 >>>>>> >>>>>>> 04:56:05,503 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/The_Way_To_Fight,1216138073477 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/The_Way_To_Fight,12161380734772008-07-18 >>>>>> >>>>>>> 04:56:05,503 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Al_Fateh-universitetet,1215959690520 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Al_Fateh-universitetet,12159596905202008-07-18 >>>>>> >>>>>>> 04:56:05,503 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://dbpedia.org/resource/Syed_Koteb,1216140016580 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Syed_Koteb,12161400165802008-07-18 >>>>>> >>>>>>> 04:56:05,503 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,1215915770009 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Humboldt-Universiteit,12159157700092008-07-18 >>>>>> >>>>>>> 04:56:05,503 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://dbpedia.org/resource/1er_RCP,1215980565468 >>>>>> 2008-07-18 < >>>>>> http://dbpedia.org/resource/1er_RCP,12159805654682008-07-18 >>>>>> >>>>>>> 04:56:05,503 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://purl.uniprot.org/taxonomy/112557,1214582151856 >>>>>> 2008-07-18< >>>>>> http://purl.uniprot.org/taxonomy/112557,12145821518562008-07-18 >>>>>> >>>>>>> 04:56:05,504 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Harry_Girvetz,1215910087698 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Harry_Girvetz,12159100876982008-07-18 >>>>>> >>>>>>> 04:56:05,504 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> http://dbpedia.org/resource/Liste_der_Autobahnen_in_Spanien,1215922127438 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Liste_der_Autobahnen_in_Spanien,12159221274382008-07-18 >>>>>> >>>>>>> 04:56:05,504 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://apassant.net/home/2007/12/flickrdf/data/groups/355509@N23,1215825925595 >>>>>> 2008-07-18< >>>>>> >>>>>> http://apassant.net/home/2007/12/flickrdf/data/groups/355509@N23,12158259255952008-07-18 >>>>>> >>>>>>> 04:56:05,504 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Category:1948_in_law,1215896373168 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Category:1948_in_law,12158963731682008-07-18 >>>>>> >>>>>>> 04:56:05,504 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://acm.rkbexplorer.com/id/person-94180,1215825860040 >>>>>> 2008-07-18< >>>>>> http://acm.rkbexplorer.com/id/person-94180,12158258600402008-07-18 >>>>>> >>>>>>> 04:56:05,504 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://www.akropolix.net/rik0/blogs/category/mac/,1215823941299 >>>>>> 2008-07-18< >>>>>> >>>>>> http://www.akropolix.net/rik0/blogs/category/mac/,12158239412992008-07-18 >>>>>> >>>>>>> 04:56:05,504 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Trevor_Ashmore,1216138073477 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Trevor_Ashmore,12161380734772008-07-18 >>>>>> >>>>>>> 04:56:05,504 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://dbpedia.org/resource/Papa_New_Guinea_Baseball_Federation,1215929139300 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Papa_New_Guinea_Baseball_Federation,12159291393002008-07-18 >>>>>> >>>>>>> 04:56:05,505 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,1215947224425 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Ursoaia%2C_Vaslui,12159472244252008-07-18 >>>>>> >>>>>>> 04:56:05,505 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,1216039960663 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Amour%2C_honneur_et_trahison,12160399606632008-07-18 >>>>>> >>>>>>> 04:56:05,505 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://dbpedia.org/resource/1960-09-03,1215980565468 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/1960-09-03,12159805654682008-07-18 >>>>>> >>>>>>> 04:56:05,505 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,1215958955911 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/A38_%28Frankrijk%29,12159589559112008-07-18 >>>>>> >>>>>>> 04:56:05,505 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,1215983091539 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dblp.rkbexplorer.com/id/people-bbf1b15b69c241e01979fb4c3d8d573f-20e639fc904f03ed0d61aa1255bc3235,12159830915392008-07-18 >>>>>> >>>>>>> 04:56:05,505 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> >>>>>> >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,1216004671842 >>>>>> 2008-07-18< >>>>>> >>>>>> http://dbpedia.org/resource/Cumbernauld_railway_station/rail_line3,12160046718422008-07-18 >>>>>> >>>>>>> 04:56:05,505 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Lysander/lang2,1215922127438 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Lysander/lang2,12159221274382008-07-18 >>>>>> >>>>>>> 04:56:05,506 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://sws.geonames.org/2701307/,1215823954578 >>>>>> 2008-07-18 < http://sws.geonames.org/2701307/,12158239545782008-07-18 >>>>>> >>>>>>> 04:56:05,506 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,1215823981137 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Bayerische_S_2/6,12158239811372008-07-18 >>>>>> >>>>>>> 04:56:05,506 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: closed >>>>>> page-repository, http://dbpedia.org/resource/Angus_McKay,1215890852482 >>>>>> 2008-07-18< >>>>>> http://dbpedia.org/resource/Angus_McKay,12158908524822008-07-18 >>>>>> >>>>>>> 04:56:05,506 >>>>>>> >>>>>> INFO org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>>> aborting server at: 140.203.154.176:60020 >>>>>> 2008-07-18 04:56:05,506 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>>> regionserver/0:0:0:0:0:0:0:0:60020 exiting >>>>>> 2008-07-18 04:56:05,561 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown >>>>>> thread. >>>>>> 2008-07-18 04:56:05,562 INFO >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread >>>>>> complete >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>> >> 2008-07-19 07:31:45,505 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer >> Exception: java.net.SocketTimeoutException: timed out waiting for rpc >> response >> 2008-07-19 07:31:45,506 WARN org.apache.hadoop.dfs.DFSClient: Error >> Recovery for block null bad datanode [0] >> 2008-07-19 07:31:54,868 FATAL org.apache.hadoop.hbase.regionserver.Flusher: >> Replay of hlog required. Forcing server restart >> org.apache.hadoop.hbase.DroppedSnapshotException >> at >> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1040) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:942) >> at >> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174) >> at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:93) >> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: >> Could not complete write to file >> /hbase/page-repository/963177038/field/info/4591605661998875281 by >> DFSClient_1009756630 >> at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:332) >> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:557) >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) >> at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source) >> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >> at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source) >> at >> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2655) >> at >> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2576) >> at >> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59) >> at >> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79) >> at >> org.apache.hadoop.hbase.regionserver.HStoreFile.writeInfo(HStoreFile.java:350) >> at >> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:747) >> at >> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:682) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1030) >> ... 3 more >> 2008-07-19 07:31:54,868 INFO org.apache.hadoop.hbase.regionserver.Flusher: >> regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting >> >> >> > >
          Hide
          Andrew Purtell added a comment -

          My response to the message in the previous comment.

          Re: Replay of hlog required, forcing regionserver restart
          Saturday, July 19, 2008 9:54 AM
          From: "Andrew Purtell" <apurtell@yahoo.com>
          To: hbase-user@hadoop.apache.org

          Hello Renaud,

          I think your cluster is overloaded.

          > WARN org.apache.hadoop.hbase.util.Sleeper: We slept 30837ms,
          > ten times longer than scheduled: 3000
          > 2008-07-18 03:55:57,783 WARN org.apache.hadoop.hbase.util.Sleeper: We
          > slept 583701ms, ten times longer than scheduled: 3000

          See https://issues.apache.org/jira/browse/HBASE-616.

          I think these types of messages imply excessive load, with processes stuck in disk wait, and/or threads starved for CPU time, etc. Processes can be stuck in disk wait while being swapped in or out also, so excessive RAM/heap is a third possibility. Given a 3 node cluster with a full complement of Hadoop and HBase daemons and 24 tasks, perhaps all three load-dimensions are redlined.

          It's hard to say what exactly was the root cause of the exception on the DFS client channel which brought down the regionserver:

          > java.io.IOException: Premeture EOF from inputStream at
          > org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)

          This could have been a socket timeout, or a fault on the data node. Does anything useful/meaningful appear in the datanode log?

          • Andy
          Show
          Andrew Purtell added a comment - My response to the message in the previous comment. Re: Replay of hlog required, forcing regionserver restart Saturday, July 19, 2008 9:54 AM From: "Andrew Purtell" <apurtell@yahoo.com> To: hbase-user@hadoop.apache.org Hello Renaud, I think your cluster is overloaded. > WARN org.apache.hadoop.hbase.util.Sleeper: We slept 30837ms, > ten times longer than scheduled: 3000 > 2008-07-18 03:55:57,783 WARN org.apache.hadoop.hbase.util.Sleeper: We > slept 583701ms, ten times longer than scheduled: 3000 See https://issues.apache.org/jira/browse/HBASE-616 . I think these types of messages imply excessive load, with processes stuck in disk wait, and/or threads starved for CPU time, etc. Processes can be stuck in disk wait while being swapped in or out also, so excessive RAM/heap is a third possibility. Given a 3 node cluster with a full complement of Hadoop and HBase daemons and 24 tasks, perhaps all three load-dimensions are redlined. It's hard to say what exactly was the root cause of the exception on the DFS client channel which brought down the regionserver: > java.io.IOException: Premeture EOF from inputStream at > org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) This could have been a socket timeout, or a fault on the data node. Does anything useful/meaningful appear in the datanode log? Andy
          Hide
          Andrew Purtell added a comment -

          And then this followup to my own response:

          Re: Replay of hlog required, forcing regionserver restart
          Saturday, July 19, 2008 10:01 AM
          From: Andrew Purtell" <apurtell@yahoo.com>
          To: hbase-user@hadoop.apache.org

          Actually I missed this:

          > 2008-07-19 07:31:45,505 WARN org.apache.hadoop.dfs.DFSClient:
          > DataStreamer Exception: java.net.SocketTimeoutException: timed out
          > waiting for rpc response

          So it was a socket timeout.

          I think you need to spread the load by adding more DFS datanodes at a minimum and probably also HBase regionservers, or reduce concurrency.

          • Andy
          Show
          Andrew Purtell added a comment - And then this followup to my own response: Re: Replay of hlog required, forcing regionserver restart Saturday, July 19, 2008 10:01 AM From: Andrew Purtell" <apurtell@yahoo.com> To: hbase-user@hadoop.apache.org Actually I missed this: > 2008-07-19 07:31:45,505 WARN org.apache.hadoop.dfs.DFSClient: > DataStreamer Exception: java.net.SocketTimeoutException: timed out > waiting for rpc response So it was a socket timeout. I think you need to spread the load by adding more DFS datanodes at a minimum and probably also HBase regionservers, or reduce concurrency. Andy
          Hide
          Andrew Purtell added a comment -

          Re: Replay of hlog required, forcing regionserver restart
          Monday, July 21, 2008 5:09 AM
          From: "Renaud Delbru" <renaud.delbru@deri.org>
          To: hbase-user@hadoop.apache.org
          [...]
          The cause of the problem was in fact the excessive load on the servers.
          Processes was stuck due to excessive swapping.
          I have reduced the number of Hadoop tasks in parallel from 24 to 12.
          Then, I have run a hadoop job for testing. There is no more rpc timeout
          errors in the logs, and HBase stays stable during the whole process (no
          regionserver went down).
          [...]

          Show
          Andrew Purtell added a comment - Re: Replay of hlog required, forcing regionserver restart Monday, July 21, 2008 5:09 AM From: "Renaud Delbru" <renaud.delbru@deri.org> To: hbase-user@hadoop.apache.org [...] The cause of the problem was in fact the excessive load on the servers. Processes was stuck due to excessive swapping. I have reduced the number of Hadoop tasks in parallel from 24 to 12. Then, I have run a hadoop job for testing. There is no more rpc timeout errors in the logs, and HBase stays stable during the whole process (no regionserver went down). [...]

            People

            • Assignee:
              Unassigned
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development