Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24454

BucketCache disabled instantly before error duration toleration is reached due to timing issue

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.10
    • Fix Version/s: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
    • Component/s: BucketCache
    • Labels:
      None
    • Environment:

      We saw this in HBase 1.4.10 (EMR 5.28.1), but I believe the newest code still has this problem.

    • Hadoop Flags:
      Reviewed

      Description

      We saw an instance where BucketCache was disabled after only two IO error were encountered at nearly the same time. The following shows all errors from a region server log for the 2020-05-26 17:00 hour. Notice that there are no other errors until the 17:14:50 at which time the BucketCache is disabled because it claims duration time has exceeded 60000 ms:

      $ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
      2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884] bucket.BucketCache: Failed reading block 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
      2020-05-26 17:14:50,397 ERROR [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0] bucket.BucketCache: Failed syncing IO engine
      2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884] bucket.BucketCache: Failed reading block 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
      2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884] bucket.BucketCache: IO errors duration time has exceeded 60000ms, disabling cache, please check your IOEngine

      The region server is very busy so it should be constantly getting successful reads and writes in the bucket cache (it is not as though there was some long ago error and then no successful IO to clear the error flag).

      We are running a busy EMR cluster backed by S3. A bucketcache getting disabled is a huge performance issue because all reads must go to S3.

      Looking at the code, I believe I've found a timing issue. Here is the code for checking and setting the ioErrorStartTime (taken from BucketCache.java):

       

        /**
         * Check whether we tolerate IO error this time. If the duration of IOEngine
         * throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable the
         * cache
         */
        private void checkIOErrorIsTolerated() {
          long now = EnvironmentEdgeManager.currentTime();
          if (this.ioErrorStartTime > 0) {
            if (cacheEnabled && (now - ioErrorStartTime) > this.ioErrorsTolerationDuration) {
              LOG.error("IO errors duration time has exceeded " + ioErrorsTolerationDuration +
                "ms, disabling cache, please check your IOEngine");
              disableCache();
            }
          } else {
            this.ioErrorStartTime = now;
          }
        }
      

       
      And here is the code for clearing the ioErrorStartTime when a successful read or write is done:

        if (this.ioErrorStartTime > 0) {
          ioErrorStartTime = -1;
        }
      

      Notice that that if ioErrorStartTime is set to -1 after the first if statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime), then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the code thinking it has exceeded ioErrorsTolerationDuration.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jacob.leblanc Jacob LeBlanc
                Reporter:
                jacob.leblanc Jacob LeBlanc
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: