Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8285

HBaseClient never recovers for single HTable.get() calls with no retries when regions move

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.94.6.1
    • Fix Version/s: 0.98.0, 0.94.7, 0.95.1
    • Component/s: Client
    • Labels:
      None

      Description

      Steps to reproduce this bug:
      1) Gracefull restart a region server causing regions to get redistributed.
      2) Client call to this region keeps failing since Meta Cache is never purged on the client for the region that moved.

      Reason behind the bug:
      1) Client continues to hit the old region server.
      2) The old region server throws NotServingRegionException which is not handled correctly and the META cache entries are never purged for that server causing the client to keep hitting the old server.

      The reason lies in ServerCallable code since we only purge META cache entries when there is a RetriesExhaustedException, SocketTimeoutException or ConnectException. However, there is no case check for NotServingRegionException(s).

      Why is this not a problem for Scan(s) and Put(s) ?

      a) If a region server is not hosting a region/scanner, then an UnknownScannerException is thrown which causes a relocateRegion() call causing a refresh of the META cache for that particular region.
      b) For put(s), the processBatchCallback() interface in HConnectionManager is used which clears out META cache entries for all kinds of exceptions except DoNotRetryException.

        Attachments

        1. 8285-trunk.txt
          1 kB
          Varun Sharma
        2. 8285-0.94.txt
          1 kB
          Varun Sharma
        3. 8285-0.94-v2.txt
          1 kB
          Varun Sharma
        4. 8285-trunk-v2.txt
          1 kB
          Varun Sharma
        5. 8285-0.94-v3.txt
          4 kB
          Varun Sharma
        6. 8285-0.94-v4.txt
          4 kB
          Varun Sharma
        7. 8285-0.94-v5.txt
          4 kB
          Varun Sharma
        8. 8285-0.94-v6.txt
          4 kB
          Varun Sharma
        9. 8285-0.95.txt
          4 kB
          Varun Sharma
        10. 8285-trunk-v3.txt
          4 kB
          Varun Sharma

          Issue Links

            Activity

              People

              • Assignee:
                varunsharma Varun Sharma
                Reporter:
                varunsharma Varun Sharma
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: