[HBASE-616] " We slept XXXXXX ms, ten times longer than scheduled: 3000" happens frequently.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 06/May/08 04:01
• Updated: 11/Jun/22 20:13
• Resolved: 08/Jul/09 20:23

Description

Just saw the below in a log... all in a row on the one server.

   4493 2008-05-05 18:08:17,512 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 34557ms, ten times longer than scheduled: 3000
   4494 2008-05-05 18:11:08,879 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 30576ms, ten times longer than scheduled: 3000
   4495 2008-05-05 18:30:45,056 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 1091720ms, ten times longer than scheduled: 3000
   4496 2008-05-05 18:30:45,056 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 1094209ms, ten times longer than scheduled: 10000
   4497 2008-05-05 18:30:45,429 FATAL org.apache.hadoop.hbase.HRegionServer: unable to report to master for 1092093 milliseconds - aborting server

We're seeing these kinda outages pretty frequently. In the case above, it was small cluster that was using TableReduce to insert. The MR, HDFS and HBase were all running on same nodes.

Issue Links

• is duplicated by: HBASE-1060 RegionServer lease expire due to LogRoller take long time and the RegionServer cannot recovery
• relates to: HBASE-412 unable to report to master
• relates to: HBASE-15 [hbase] Could not complete hdfs write out to flush file forcing regionserver restart