Details

    • Sub-task
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 1.3.1
    • 1.3.2, 1.4.1, 2.0.0
    • regionserver, Scanners
    • None
    • Reviewed

    Description

      We see FNFE exceptions on our 1.3 clusters when scans and flushes happen at the same time. This causes regionserver to throw a UnknownScannerException and client retries.

      This happens during the following sequence:

      1. Scanner open, client fetched some rows from regionserver and working on it
      2. Flush happens and storeScanner is updated with flushed files (StoreScanner.updateReaders())
      3. Compaction happens on the region while scanner is still open
      4. compaction discharger runs and cleans up the newly flushed file as we don't have new scanners on it yet.
      5. Client issues scan.next and during StoreScanner.resetScannerStack(), we get a FNFE. RegionServer throws a UnknownScannerThe client retries in 1.3. With branch-1.4, the scan fails with a DoNotRetryIOException.

      ram_krish, My proposal is to increment the reader count during updateReaders() and decrement it during resetScannerStack(), so discharger doesn't clean it up. Scan lease expiries also have to be taken care of. Am I missing anything? Is there a better approach?

      Attachments

        1. HBASE-19468-poc.patch
          8 kB
          Thiruvel Thirumoolan
        2. HBASE-19468_master.patch
          6 kB
          ramkrishna.s.vasudevan
        3. HBASE-19468_1.4.patch
          9 kB
          ramkrishna.s.vasudevan

        Issue Links

          Activity

            People

              ram_krish ramkrishna.s.vasudevan
              thiruvel Thiruvel Thirumoolan
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: