Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HCD already talks of in-memory columns; its just not implemented. With hfile this should now be possible – if set, read in the whole storefile and serve from the blockbuffer.

      1. HBASE-1218-v1.patch
        24 kB
        Jonathan Gray
      2. HBASE-1218-v2.patch
        18 kB
        Jonathan Gray
      3. HBASE-1218-v3.patch
        19 kB
        Jonathan Gray

        Issue Links

          Activity

          Hide
          Jonathan Gray added a comment -

          In-memory column families can be implemented with a modification to the BlockCache LRU Map. Will require an additional parameter to methods interacting with the blockcache map (whether block comes from in-memory family or not).

          Show
          Jonathan Gray added a comment - In-memory column families can be implemented with a modification to the BlockCache LRU Map. Will require an additional parameter to methods interacting with the blockcache map (whether block comes from in-memory family or not).
          Hide
          Jonathan Gray added a comment -

          Will be enabled by HBASE-1460. Will still require a patch to wire things up after 1460 goes in

          Show
          Jonathan Gray added a comment - Will be enabled by HBASE-1460 . Will still require a patch to wire things up after 1460 goes in
          Hide
          Jonathan Gray added a comment -

          The LRU in trunk now supports in-memory blocks.

          In HFile.Reader.readBlock(int) we need to change cache.cacheBlock(blockName, blockBuf) to cache.cacheBlock(blockName, blockBuf, boolean inMemory).

          HFile.Reader gets instantiated from within a Store so should be easy enough to pass something in from the family configuration there.

          I strongly recommend lazy-loading, no pre-fetching of blocks. That's how they do it in big table. If you want to warm the blocks, just scan the table.

          Show
          Jonathan Gray added a comment - The LRU in trunk now supports in-memory blocks. In HFile.Reader.readBlock(int) we need to change cache.cacheBlock(blockName, blockBuf) to cache.cacheBlock(blockName, blockBuf, boolean inMemory). HFile.Reader gets instantiated from within a Store so should be easy enough to pass something in from the family configuration there. I strongly recommend lazy-loading, no pre-fetching of blocks. That's how they do it in big table. If you want to warm the blocks, just scan the table.
          Hide
          Jonathan Gray added a comment -

          Passes a boolean in from Store to StoreFile to HFile.Reader and then to the LRU block cache.

          Not much to it.

          This patch incorporates some of the changes from HBASE-1597. I will remake a patch after that gets committed, but this is up for review.

          Show
          Jonathan Gray added a comment - Passes a boolean in from Store to StoreFile to HFile.Reader and then to the LRU block cache. Not much to it. This patch incorporates some of the changes from HBASE-1597 . I will remake a patch after that gets committed, but this is up for review.
          Hide
          stack added a comment -

          Chatting with Jon, this patch needs more. In-memory is a table setting. Rare will be case where we want a table in memory. Jon is going to make it so in-memory is a column setting. I'll take care of migration part over in 1295.

          Show
          stack added a comment - Chatting with Jon, this patch needs more. In-memory is a table setting. Rare will be case where we want a table in memory. Jon is going to make it so in-memory is a column setting. I'll take care of migration part over in 1295.
          Hide
          Jonathan Gray added a comment -

          post 1597 patch

          Show
          Jonathan Gray added a comment - post 1597 patch
          Hide
          Jonathan Gray added a comment -

          Ready for final review and commit.

          Show
          Jonathan Gray added a comment - Ready for final review and commit.
          Hide
          Jonathan Gray added a comment -

          Adds better logging at the end of evictions, outputting the sizes of the three priorities of blocks (single, multiple, memory)

          Show
          Jonathan Gray added a comment - Adds better logging at the end of evictions, outputting the sizes of the three priorities of blocks (single, multiple, memory)
          Hide
          stack added a comment -

          Committed. Thanks for patch Jon.

          Show
          stack added a comment - Committed. Thanks for patch Jon.

            People

            • Assignee:
              Jonathan Gray
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development