HBase
  1. HBase
  2. HBASE-9840

Large scans and BlockCache evictions problems

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Later
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I just ran into a scenario that baffled me first, but after some reflection makes sense. I ran a very large scan that filled up most of the block cache with my scan's data. I ran that scan a few times.

      That I ran a much smaller scan, and this scan will never get all its blocks cached if it does not fit entirely into the remaining BlockCache; regardless how I often I run it!

      The reason is that the blocks of the first large scan were all promoted. Since the 2nd scan did not fully fit into the cache all blocks are round-robin evicted as I rerun the scan. Thus those blocks will never get accessed more than once before they get evicted again.

      Since promoted blocks are not demoted the large scan's block will never be evicted unless we have another small enough scan/get that can promote its blocks.

      Not sure what the proper solution is, but it seems only a LRU cache that can expire blocks over time would solve this.

      Granted, this is a pretty special case.

      Edit: My usual spelling digressions.

        Activity

        Hide
        Vladimir Rodionov added a comment -

        What is the purpose of running large scan (larger than available young gen cache space) with cacheBlock enabled?

        I do have a couple comments on LruBlockCache design and implementation. I did not spend much time analyzing the code though:

        1. singleFactor = 0.25, multiFactor = 0.5 , memoryFactor = 0.25 - are all hard-coded. They must be configurable.
        2. Default values are probably not optimal at all. If it is attempt to mimic LRU2Q cache than optimal split for first insert is closer to 0.75 (I think I read it on Facebook engineering)
        3. Eviction does not follow LRU2Q at all. In LRU2Q all data is evicted from the tail of a common queue (say, single and multi bucket CAN NOT be processed separately but together). In LruBlockCache all 3 buckets evict their data independently.

        Show
        Vladimir Rodionov added a comment - What is the purpose of running large scan (larger than available young gen cache space) with cacheBlock enabled? I do have a couple comments on LruBlockCache design and implementation. I did not spend much time analyzing the code though: 1. singleFactor = 0.25, multiFactor = 0.5 , memoryFactor = 0.25 - are all hard-coded. They must be configurable. 2. Default values are probably not optimal at all. If it is attempt to mimic LRU2Q cache than optimal split for first insert is closer to 0.75 (I think I read it on Facebook engineering) 3. Eviction does not follow LRU2Q at all. In LRU2Q all data is evicted from the tail of a common queue (say, single and multi bucket CAN NOT be processed separately but together). In LruBlockCache all 3 buckets evict their data independently.
        Hide
        Vladimir Rodionov added a comment -

        By the way, Lars, 3. would solve the issue you have observed.

        Show
        Vladimir Rodionov added a comment - By the way, Lars, 3. would solve the issue you have observed.
        Hide
        Lars Hofhansl added a comment -

        Normally you'd not cache your blocks on large scan (that's why "this is a pretty special case").
        Yeah #3 would fix this.

        This is not immediately important, just a note that our caching needs work

        Show
        Lars Hofhansl added a comment - Normally you'd not cache your blocks on large scan (that's why "this is a pretty special case"). Yeah #3 would fix this. This is not immediately important, just a note that our caching needs work
        Hide
        Jean-Daniel Cryans added a comment -

        singleFactor = 0.25, multiFactor = 0.5 , memoryFactor = 0.25 - are all hard-coded. They must be configurable.

        Meh, +0, see the following.

        Default values are probably not optimal at all. If it is attempt to mimic LRU2Q cache than optimal split for first insert is closer to 0.75 (I think I read it on Facebook engineering)

        From its javadoc:

        Each priority will retain close to its maximum size, however, if any priority is not using its entire chunk the others are able to grow beyond their chunk size.

        So if you aren't using the in-memory part, you effectively dedicate 75% of your BC to the multi priority.

        Eviction does not follow LRU2Q at all

        2Q suffers from different problems, mainly the tuning of Kin and Kout.

        The "best" page replacement policy is one that takes into account age and frequency of access, but we can't use ARC.

        Show
        Jean-Daniel Cryans added a comment - singleFactor = 0.25, multiFactor = 0.5 , memoryFactor = 0.25 - are all hard-coded. They must be configurable. Meh, +0, see the following. Default values are probably not optimal at all. If it is attempt to mimic LRU2Q cache than optimal split for first insert is closer to 0.75 (I think I read it on Facebook engineering) From its javadoc: Each priority will retain close to its maximum size, however, if any priority is not using its entire chunk the others are able to grow beyond their chunk size. So if you aren't using the in-memory part, you effectively dedicate 75% of your BC to the multi priority. Eviction does not follow LRU2Q at all 2Q suffers from different problems, mainly the tuning of Kin and Kout. The "best" page replacement policy is one that takes into account age and frequency of access, but we can't use ARC.
        Hide
        Vladimir Rodionov added a comment -

        When all three buckets become full all new blocks are inserted at 0.25 split in a queue. Always, because :

        1. cache entries never expire.
        2. all buckets manage eviction independently.

        Show
        Vladimir Rodionov added a comment - When all three buckets become full all new blocks are inserted at 0.25 split in a queue. Always, because : 1. cache entries never expire. 2. all buckets manage eviction independently.
        Hide
        Jean-Daniel Cryans added a comment -

        When all three buckets become full all new blocks are inserted at 0.25 split in a queue

        Yes, but like I am saying 2Q has its own configs that you can't change at runtime and you will hit another edgy use case that isn't able to use all the cache.

        Show
        Jean-Daniel Cryans added a comment - When all three buckets become full all new blocks are inserted at 0.25 split in a queue Yes, but like I am saying 2Q has its own configs that you can't change at runtime and you will hit another edgy use case that isn't able to use all the cache.

          People

          • Assignee:
            Unassigned
            Reporter:
            Lars Hofhansl
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development