Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9902

Region Server is starting normally even if clock skew is more than default 30 seconds(or any configured). -> Regionserver node time is greater than master node time

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.11
    • Fix Version/s: 0.98.0, 0.96.1, 0.94.14
    • Component/s: master
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Clock skew detection to be made absolute value comparison. Any time difference between master or region, high or low must prevent the region server startup

      Description

      When Region server's time is ahead of Master's time and the difference is more than hbase.master.maxclockskew value, region server startup is not failing with ClockOutOfSyncException.
      This causes some abnormal behavior as detected by our Tests.

      ServerManager.java#checkClockSkew
      long skew = System.currentTimeMillis() - serverCurrentTime;
      if (skew > maxSkew)

      { String message = "Server " + serverName + " has been " + "rejected; Reported time is too far out of sync with master. " + "Time difference of " + skew + "ms > max allowed of " + maxSkew + "ms"; LOG.warn(message); throw new ClockOutOfSyncException(message); }

      Above line results in negative value when Master's time is lesser than region server time and " if (skew > maxSkew) " check fails to find the skew in this case.

      Please Note: This was tested in hbase 0.94.11 version and the trunk also currently has the same logic.

      The fix for the same would be to make the skew positive value first as below:

      long skew = System.currentTimeMillis() - serverCurrentTime;
      skew = (skew < 0 ? -skew : skew);
      if (skew > maxSkew) {.....

        Attachments

        1. HBASE-9902_v2-0.94.patch
          3 kB
          Kashif
        2. HBASE-9902_v2.patch
          3 kB
          Kashif
        3. HBASE-9902.patch
          3 kB
          Kashif

          Activity

            People

            • Assignee:
              kashifjs@huawei.com Kashif
              Reporter:
              kashifjs@huawei.com Kashif
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: