Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-13082

Coarsen StoreScanner locks to RegionScanner

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.3.0, 2.0.0
    • Performance, Scanners
    • None
    • Reviewed
    • Hide
      After this JIRA we will not be doing any scanner reset after compaction during a course of a scan. The files that were compacted will still be continued to be used in the scan process. The compacted files will be archived by a background thread that runs every 2 mins by default only when there are no active scanners on those comapcted files. The above duration can be controlled using the knob 'hbase.hfile.compactions.cleaner.interval'.
      Show
      After this JIRA we will not be doing any scanner reset after compaction during a course of a scan. The files that were compacted will still be continued to be used in the scan process. The compacted files will be archived by a background thread that runs every 2 mins by default only when there are no active scanners on those comapcted files. The above duration can be controlled using the knob 'hbase.hfile.compactions.cleaner.interval'.

    Description

      Continuing where HBASE-10015 left of.
      We can avoid locking (and memory fencing) inside StoreScanner by deferring to the lock already held by the RegionScanner.
      In tests this shows quite a scan improvement and reduced CPU (the fences make the cores wait for memory fetches).

      There are some drawbacks too:

      • All calls to RegionScanner need to be remain synchronized
      • Implementors of coprocessors need to be diligent in following the locking contract. For example Phoenix does not lock RegionScanner.nextRaw() and required in the documentation (not picking on Phoenix, this one is my fault as I told them it's OK)
      • possible starving of flushes and compaction with heavy read load. RegionScanner operations would keep getting the locks and the flushes/compactions would not be able finalize the set of files.

      I'll have a patch soon.

      Attachments

        1. CountDownLatch-0.98.txt
          6 kB
          Lars Hofhansl
        2. HBASE-13082_19.patch
          108 kB
          ramkrishna.s.vasudevan
        3. HBASE-13082_18.patch
          106 kB
          ramkrishna.s.vasudevan
        4. HBASE-13082_17.patch
          105 kB
          ramkrishna.s.vasudevan
        5. HBASE-13082_16.patch
          104 kB
          ramkrishna.s.vasudevan
        6. HBASE-13082_15.patch
          107 kB
          ramkrishna.s.vasudevan
        7. LockVsSynchronized.java
          2 kB
          ramkrishna.s.vasudevan
        8. HBASE-13082_withpatch.jpg
          93 kB
          ramkrishna.s.vasudevan
        9. HBASE-13082_withoutpatch.jpg
          29 kB
          ramkrishna.s.vasudevan
        10. HBASE-13082_14.patch
          107 kB
          ramkrishna.s.vasudevan
        11. HBASE-13082_13.patch
          106 kB
          ramkrishna.s.vasudevan
        12. HBASE-13082_12.patch
          106 kB
          ramkrishna.s.vasudevan
        13. HBASE-13082_2.pdf
          228 kB
          ramkrishna.s.vasudevan
        14. HBASE-13082_1.pdf
          229 kB
          ramkrishna.s.vasudevan
        15. HBASE-13082_9.patch
          100 kB
          ramkrishna.s.vasudevan
        16. HBASE-13082_9.patch
          101 kB
          ramkrishna.s.vasudevan
        17. HBASE-13082.pdf
          116 kB
          ramkrishna.s.vasudevan
        18. HBASE-13082_4.patch
          148 kB
          ramkrishna.s.vasudevan
        19. HBASE-13082_3.patch
          119 kB
          ramkrishna.s.vasudevan
        20. HBASE-13082_2_WIP.patch
          99 kB
          ramkrishna.s.vasudevan
        21. HBASE-13082_1_WIP.patch
          100 kB
          ramkrishna.s.vasudevan
        22. 13082-v4.txt
          20 kB
          Lars Hofhansl
        23. 13082-v3.txt
          17 kB
          Lars Hofhansl
        24. 13082-v2.txt
          9 kB
          Lars Hofhansl
        25. gc.png
          20 kB
          Michael Stack
        26. hits.png
          13 kB
          Michael Stack
        27. next.png
          14 kB
          Michael Stack
        28. gc.png
          22 kB
          Michael Stack
        29. gc.png
          24 kB
          Michael Stack
        30. next.png
          14 kB
          Michael Stack
        31. 13082.txt
          8 kB
          Lars Hofhansl
        32. 13082-test.txt
          3 kB
          Lars Hofhansl
        33. 13082.txt
          8 kB
          Lars Hofhansl

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ram_krish ramkrishna.s.vasudevan
            larsh Lars Hofhansl
            Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment