[HBASE-2606] NativeException: org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in .META. for region item,,1274235159640

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.4
• Fix Version/s: 0.20.5

Environment

Linux x86_64, java 1.6.0_20, hbase 0.20.4, CDH3

• Assignee: Unassigned
• Reporter: Sam Pullara
• Created: 24/May/10 00:32
• Updated: 12/Oct/12 06:15
• Resolved: 25/May/10 00:28

Description

Though the region is listed when you attempt to access it you get an error that it is not assigned to a region. In the log you find this repeating over and over:

2010-05-23 16:10:08,958 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 5.0
2010-05-23 16:10:09,519 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region

{server: 10.211.55.4:60020, regionname: -ROOT-,,0, startKey: <>}

2010-05-23 16:10:09,529 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1 row(s) of meta region

{server: 10.211.55.4:60020, regionname: -ROOT-,,0, startKey: <>}

complete
2010-05-23 16:10:52,178 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region

{server: 10.211.55.4:60020, regionname: .META.,,1, startKey: <>}

2010-05-23 16:10:52,188 DEBUG org.apache.hadoop.hbase.master.BaseScanner: Current assignment of item,,1274235159640 is not valid; serverAddress=, startCode=0 unknown.
2010-05-23 16:10:52,193 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of 3 row(s) of meta region

{server: 10.211.55.4:60020, regionname: .META.,,1, startKey: <>}

complete
2010-05-23 16:10:52,193 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned
2010-05-23 16:10:53,139 INFO org.apache.hadoop.hbase.master.RegionManager: Assigning region item,,1274235159640 to hbase,60020,1274655730505
2010-05-23 16:10:53,143 INFO org.apache.hadoop.hbase.master.ServerManager: Processing MSG_REPORT_OPEN: item,,1274235159640 from hbase,60020,1274655730505; 1 of 1
2010-05-23 16:10:53,143 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: PendingOpenOperation from hbase,60020,1274655730505
2010-05-23 16:10:53,143 INFO org.apache.hadoop.hbase.master.RegionServerOperation: item,,1274235159640 open on 10.211.55.4:60020
2010-05-23 16:10:53,146 INFO org.apache.hadoop.hbase.master.RegionServerOperation: Updated row item,,1274235159640 in region .META.,,1 with startcode=1274655730505, server=10.211.55.4:60020

Doing a flush and a major_compaction of the table resets it. I have a VM snapshot that has the behavior and it is reproducible at will. Attached are the log files from the .META. region at jdcryans request.