[HBASE-1121] Cluster confused about where -ROOT- is

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.19.1, 0.20.0

Environment

TRUNK trying to do big upload

• Assignee: Jim Kellerman
• Reporter: Michael Stack
• Created: 11/Jan/09 22:44
• Updated: 13/Sep/09 22:24
• Resolved: 23/Jan/09 23:24

Description

Each node in cluster is doing below:

2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
        at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
>. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
        at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)

I tried to close the root region and tripped over hbase-1120.

So I killed the regionserver hosting ROOT but weirdly had no effect as though the regionservers' internal cache of ROOT location cannot be changed.