Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5930

Limits the amount of time an edit can live in the memstore.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.98.0, 0.94.8, 0.95.1
    • None
    • None
    • Reviewed
    • Hide
      This feature limits the time that unflushed data will stay in the memstore.
      By default a memstore will flush if data older than 1h (3600000ms) is present.

      This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 3600000).
      Show
      This feature limits the time that unflushed data will stay in the memstore. By default a memstore will flush if data older than 1h (3600000ms) is present. This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 3600000).

    Description

      A colleague of mine ran into an interesting issue.
      He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory.

      Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks.

      Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc.

      Attachments

        1. 5930-wip.patch
          10 kB
          Devaraj Das
        2. 5930-1.patch
          10 kB
          Devaraj Das
        3. 5930-2.1.patch
          13 kB
          Devaraj Das
        4. 5930-2.2.patch
          13 kB
          Devaraj Das
        5. 5930-2.3.patch
          17 kB
          Devaraj Das
        6. 5930-2.4.patch
          17 kB
          Devaraj Das
        7. 5930-track-oldest-sample.txt
          14 kB
          Lars Hofhansl
        8. 5930-0.94.txt
          14 kB
          Lars Hofhansl
        9. HBASE-5930-ADD-0.patch
          2 kB
          Elliott Neil Clark
        10. hbase-5930-addendum2.patch
          0.7 kB
          Enis Soztutar
        11. hbase-5930-test-execution.log
          18 kB
          Enis Soztutar
        12. 5930-0.94-added-addendum.txt
          15 kB
          Devaraj Das
        13. 5930-addendum-for-disabling.trunk.txt
          1 kB
          Devaraj Das
        14. 5930-0.94-2.txt
          15 kB
          Devaraj Das
        15. 5930-addendum-for-disabling.trunk.with-tests.txt
          7 kB
          Devaraj Das
        16. 5930-addendum-for-disabling.trunk.with-tests.txt
          7 kB
          Devaraj Das
        17. 5930-0.94-added-addendum-with-tests.txt
          20 kB
          Devaraj Das

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-3707 Flush memstore after a configurable number of inserts not simply based on size

          • Major - Major loss of function.
          • Closed
          is required by

          Umbrella - An overarching type made of sub-tasks HBASE-5843 Improve HBase MTTR - Mean Time To Recover

          • Major - Major loss of function.
          • Closed

          Activity

            People

              ddas Devaraj Das
              larsh Lars Hofhansl
              Votes:
              1 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: