[HBASE-3168] Sanity date and time check when a region server joins the cluster

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.89.20100924
• Fix Version/s: 0.90.0
• Component/s: regionserver

Environment

RHEL 5.5 64bit, 1 Master 4 Region Servers

• Hadoop Flags: Reviewed
• Assignee: Jeff Whiting
• Reporter: Jeff Whiting
• Created: 29/Oct/10 17:56
• Updated: 12/Oct/12 06:17
• Resolved: 10/Nov/10 04:55

Description

Introduce a sanity check when a RS joins the cluster to make sure its clock isn't too far out of skew with the rest of the cluster. If the RS's time is too far out of skew then the master would prevent it from joining and RS would die and log the error.

Having a RS with even small differences in time can cause huge problems due to how bhase stores values with timestamps.

According to J-D in ServerManager we are already doing:

    HServerInfo info = new HServerInfo(serverInfo);
    checkIsDead(info.getServerName(), "STARTUP");
    checkAlreadySameHostPort(info);
    recordNewServer(info, false, null);

And that the new check would fit in nicely there.

JG suggests we add a "ClockOutOfSync-like exception"