HBase
  1. HBase
  2. HBASE-10418

give blocks of smaller store files priority in cache

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None

      Description

      That's just an idea at this point, I don't have a patch nor plan to make one in near future.
      It's good for datasets that don't fit in memory especially; and if scans are involved.
      Scans (and gets in absence of bloom filters' help) have to read from all store files. Short range request will hit one block in every file.
      If small files are more likely to be entirely available in memory, on average requests will hit less blocks from FS.
      For scans that read a lot of data, it's better to read blocks in sequence from a big file and blocks for small files from cache, rather than a mix of FS and cached blocks from different files, because the (HBase) blocks of a big file would be sequential in one HDFS block.

        Activity

        Hide
        Andrew Purtell added a comment -

        This presumes that smaller HFiles contain the data of interest for short scans. What kind of mechanism to we have in place to make that more likely than not?

        Would it be better to do a bit of schema design such that small files / short scan data is segregated to one column family and the large files / large scan data to another, and then prioritize in cache by column family?

        Show
        Andrew Purtell added a comment - This presumes that smaller HFiles contain the data of interest for short scans. What kind of mechanism to we have in place to make that more likely than not? Would it be better to do a bit of schema design such that small files / short scan data is segregated to one column family and the large files / large scan data to another, and then prioritize in cache by column family?
        Hide
        Sergey Shelukhin added a comment -

        If you have such knowledge yes. I am talking about unknown data distribution, within the same table/cf for simplicity.
        First, if compactions happen in normal pattern, we'll have a large file from major compaction and small files from flushes/minors.
        If we don't know the data distribution, what is described above would be the expected pattern...
        Specifically for scans, they cannot use bloom filters, and pretty much have to hit a block of each file, no matter the data distribution, right?

        Show
        Sergey Shelukhin added a comment - If you have such knowledge yes. I am talking about unknown data distribution, within the same table/cf for simplicity. First, if compactions happen in normal pattern, we'll have a large file from major compaction and small files from flushes/minors. If we don't know the data distribution, what is described above would be the expected pattern... Specifically for scans, they cannot use bloom filters, and pretty much have to hit a block of each file, no matter the data distribution, right?
        Hide
        Sergey Shelukhin added a comment -

        Unless file key range and scan key range don't intersect

        Show
        Sergey Shelukhin added a comment - Unless file key range and scan key range don't intersect

          People

          • Assignee:
            Unassigned
            Reporter:
            Sergey Shelukhin
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development