[HBASE-25] [hbase] Stuck regionserver?

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Invalid
• Fix Version/s: 0.1.0
• Component/s: regionserver
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 04/Dec/07 03:37
• Updated: 22/Aug/08 21:17
• Resolved: 14/Mar/08 03:44

Description

Looking in logs, a regionserver went down because it could not contact the master after 60 seconds. Watching logging, the HRS is repeatedly checking all 150 loaded regions over and over again w/ a pause of about 5 seconds between runs... then there is a suspicious 60+ second gap with no logging as though the regionserver had hung up on something:

2007-12-03 13:14:54,178 DEBUG hbase.HRegionServer - flushing region postlog,img151/60/plakatlepperduzy1hh7.jpg,1196614355635
2007-12-03 13:14:54,178 DEBUG hbase.HRegion - Not flushing cache for region postlog,img151/60/plakatlepperduzy1hh7.jpg,1196614355635: snapshotMemcaches() determined that there was nothing to do
2007-12-03 13:14:54,205 DEBUG hbase.HRegionServer - flushing region postlog,img247/230/seanpaul4li.jpg,1196615889965
2007-12-03 13:14:54,205 DEBUG hbase.HRegion - Not flushing cache for region postlog,img247/230/seanpaul4li.jpg,1196615889965: snapshotMemcaches() determined that there was nothing to do
2007-12-03 13:16:04,305 FATAL hbase.HRegionServer - unable to report to master for 67467 milliseconds - aborting server
2007-12-03 13:16:04,455 INFO  hbase.Leases - regionserver/0:0:0:0:0:0:0:0:60020 closing leases
2007-12-03 13:16:04,455 INFO  hbase.Leases$LeaseMonitor - regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker exiting

Master seems to be running fine scanning its ~700 regions. Then you see this in log, before the HRS shuts itself down.

2007-12-03 13:14:31,416 INFO  hbase.Leases - HMaster.leaseChecker lease expired 153260899/1532608992007-12-03 13:14:31,417 INFO  hbase.HMaster - XX.XX.XX.102:60020 lease expired

... and we go on to process shutdown.