HBase
  1. HBase
  2. HBASE-5930

Limits the amount of time an edit can live in the memstore.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.94.8, 0.95.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This feature limits the time that unflushed data will stay in the memstore.
      By default a memstore will flush if data older than 1h (3600000ms) is present.

      This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 3600000).
      Show
      This feature limits the time that unflushed data will stay in the memstore. By default a memstore will flush if data older than 1h (3600000ms) is present. This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 3600000).

      Description

      A colleague of mine ran into an interesting issue.
      He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory.

      Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks.

      Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc.

      1. hbase-5930-test-execution.log
        18 kB
        Enis Soztutar
      2. hbase-5930-addendum2.patch
        0.7 kB
        Enis Soztutar
      3. HBASE-5930-ADD-0.patch
        2 kB
        Elliott Clark
      4. 5930-wip.patch
        10 kB
        Devaraj Das
      5. 5930-track-oldest-sample.txt
        14 kB
        Lars Hofhansl
      6. 5930-addendum-for-disabling.trunk.with-tests.txt
        7 kB
        Devaraj Das
      7. 5930-addendum-for-disabling.trunk.with-tests.txt
        7 kB
        Devaraj Das
      8. 5930-addendum-for-disabling.trunk.txt
        1 kB
        Devaraj Das
      9. 5930-2.4.patch
        17 kB
        Devaraj Das
      10. 5930-2.3.patch
        17 kB
        Devaraj Das
      11. 5930-2.2.patch
        13 kB
        Devaraj Das
      12. 5930-2.1.patch
        13 kB
        Devaraj Das
      13. 5930-1.patch
        10 kB
        Devaraj Das
      14. 5930-0.94-added-addendum-with-tests.txt
        20 kB
        Devaraj Das
      15. 5930-0.94-added-addendum.txt
        15 kB
        Devaraj Das
      16. 5930-0.94-2.txt
        15 kB
        Devaraj Das
      17. 5930-0.94.txt
        14 kB
        Lars Hofhansl

        Issue Links

          Activity

          Lars Hofhansl created issue -
          Lars Hofhansl made changes -
          Field Original Value New Value
          Link This issue is related to HBASE-3707 [ HBASE-3707 ]
          Nicolas Liochon made changes -
          Link This issue is required by HBASE-5843 [ HBASE-5843 ]
          Devaraj Das made changes -
          Assignee Devaraj Das [ devaraj ]
          Devaraj Das made changes -
          Attachment 5930-wip.patch [ 12566224 ]
          Devaraj Das made changes -
          Fix Version/s 0.96.0 [ 12320040 ]
          Devaraj Das made changes -
          Attachment 5930-1.patch [ 12566262 ]
          Devaraj Das made changes -
          Attachment 5930-2.1.patch [ 12566533 ]
          Devaraj Das made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Devaraj Das made changes -
          Attachment 5930-2.2.patch [ 12566881 ]
          Devaraj Das made changes -
          Attachment 5930-2.3.patch [ 12568108 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          stack made changes -
          Fix Version/s 0.95.1 [ 12324288 ]
          Fix Version/s 0.95.0 [ 12324094 ]
          Devaraj Das made changes -
          Attachment 5930-2.4.patch [ 12580196 ]
          Lars Hofhansl made changes -
          Attachment 5930-track-oldest-sample.txt [ 12580468 ]
          Devaraj Das made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          Lars Hofhansl made changes -
          Summary Periodically flush the Memstore? Limits the amount of time an edit can live in the memstore.
          Lars Hofhansl made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.8 [ 12324145 ]
          Lars Hofhansl made changes -
          Attachment 5930-0.94.txt [ 12580577 ]
          Elliott Clark made changes -
          Attachment HBASE-5930-ADD-0.patch [ 12580587 ]
          Enis Soztutar made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Enis Soztutar made changes -
          Attachment hbase-5930-addendum2.patch [ 12580601 ]
          Attachment hbase-5930-test-execution.log [ 12580602 ]
          Devaraj Das made changes -
          Attachment 5930-0.94-added-addendum.txt [ 12581723 ]
          Lars Hofhansl made changes -
          Release Note This feature limits the time that unflushed data will stay in the memstore.
          By default a memstore will flush if data older than 1h (3600000ms) is present.

          This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 3600000).
          Devaraj Das made changes -
          Attachment 5930-addendum-for-disabling.trunk.txt [ 12582788 ]
          Attachment 5930-0.94-2.txt [ 12582789 ]
          Devaraj Das made changes -
          Devaraj Das made changes -
          Devaraj Das made changes -
          Attachment 5930-0.94-added-addendum-with-tests.txt [ 12583054 ]
          Lars Hofhansl made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          yang ming made changes -
          Environment 11
          yang ming made changes -
          Environment 11
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Devaraj Das
              Reporter:
              Lars Hofhansl
            • Votes:
              1 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development