Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24749 Direct insert HFiles and Persist in-memory HFile tracking
  3. HBASE-25391

Flush directly into data directory, skip rename when committing flush

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None
    • Hide
      This adds a new StoreFlushContext implementation, PersistedStoreFlushContext, together with a new StoreFlusher implementation, PersistedEngineStoreFlusher. As the memstore flush process is actually comprised of an initial file creations, followed by the actual write and finally a "commit" action, these two should be always be used together to guarantee a successful flush that doesn't involve creating temp files that get renamed later at the "commit" stage. By setting PersistedEngineStoreFlusher as the StoreFlusher implementation at "hbase.hstore.defaultengine.storeflusher.class" configuration, memstore flushes will create the resulting hfile directly in the store dir, instead of using a temp dir. Complementing the flush, PersistedStoreFlushContext, configured by hbase.regionserver.store.flush.context.class, assumes committed store files were written directly in the store dir, and therefore, doesn't perform a rename from tmp dir into the store dir.

      Note: This requires, specific StoreEngine and StoreFileManager implementations capable of tracking committed from non-committed files, like the ones implemented by HBASE-25395.
      Show
      This adds a new StoreFlushContext implementation, PersistedStoreFlushContext, together with a new StoreFlusher implementation, PersistedEngineStoreFlusher. As the memstore flush process is actually comprised of an initial file creations, followed by the actual write and finally a "commit" action, these two should be always be used together to guarantee a successful flush that doesn't involve creating temp files that get renamed later at the "commit" stage. By setting PersistedEngineStoreFlusher as the StoreFlusher implementation at "hbase.hstore.defaultengine.storeflusher.class" configuration, memstore flushes will create the resulting hfile directly in the store dir, instead of using a temp dir. Complementing the flush, PersistedStoreFlushContext, configured by hbase.regionserver.store.flush.context.class, assumes committed store files were written directly in the store dir, and therefore, doesn't perform a rename from tmp dir into the store dir. Note: This requires, specific StoreEngine and StoreFileManager implementations capable of tracking committed from non-committed files, like the ones implemented by HBASE-25395 .

    Description

      When flushing memstore snapshot to HFile, we write it directly to the data directory.

      Attachments

        Issue Links

          is required by

          Sub-task - The sub-task of the issue HBASE-25396 Integration with direct insert into data/cf directory

          • Major - Major loss of function.
          • Closed
          requires

          Sub-task - The sub-task of the issue HBASE-25907 Move StoreFlushContext out of HStore and make it pluggable

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #3354

          Activity

            People

              wchevreuil Wellington Chevreuil
              taklwu Tak-Lon (Stephen) Wu
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: