HBase
  1. HBase
  2. HBASE-5930

Limits the amount of time an edit can live in the memstore.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.94.8, 0.95.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This feature limits the time that unflushed data will stay in the memstore.
      By default a memstore will flush if data older than 1h (3600000ms) is present.

      This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 3600000).
      Show
      This feature limits the time that unflushed data will stay in the memstore. By default a memstore will flush if data older than 1h (3600000ms) is present. This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 3600000).

      Description

      A colleague of mine ran into an interesting issue.
      He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory.

      Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks.

      Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc.

      1. hbase-5930-test-execution.log
        18 kB
        Enis Soztutar
      2. hbase-5930-addendum2.patch
        0.7 kB
        Enis Soztutar
      3. HBASE-5930-ADD-0.patch
        2 kB
        Elliott Clark
      4. 5930-wip.patch
        10 kB
        Devaraj Das
      5. 5930-track-oldest-sample.txt
        14 kB
        Lars Hofhansl
      6. 5930-addendum-for-disabling.trunk.with-tests.txt
        7 kB
        Devaraj Das
      7. 5930-addendum-for-disabling.trunk.with-tests.txt
        7 kB
        Devaraj Das
      8. 5930-addendum-for-disabling.trunk.txt
        1 kB
        Devaraj Das
      9. 5930-2.4.patch
        17 kB
        Devaraj Das
      10. 5930-2.3.patch
        17 kB
        Devaraj Das
      11. 5930-2.2.patch
        13 kB
        Devaraj Das
      12. 5930-2.1.patch
        13 kB
        Devaraj Das
      13. 5930-1.patch
        10 kB
        Devaraj Das
      14. 5930-0.94-added-addendum-with-tests.txt
        20 kB
        Devaraj Das
      15. 5930-0.94-added-addendum.txt
        15 kB
        Devaraj Das
      16. 5930-0.94-2.txt
        15 kB
        Devaraj Das
      17. 5930-0.94.txt
        14 kB
        Lars Hofhansl

        Issue Links

          Activity

          Lars Hofhansl created issue -
          Hide
          Todd Lipcon added a comment -

          Seems reasonable to flush the memstore if it's had no write activity at all in N minutes. Then it shouldn't lead to smaller storefiles, right?

          Show
          Todd Lipcon added a comment - Seems reasonable to flush the memstore if it's had no write activity at all in N minutes. Then it shouldn't lead to smaller storefiles, right?
          Hide
          Lars Hofhansl added a comment -

          What should trigger the flush is an interesting discussion in itself. Should we flush:

          • after N timeunits of write inactivity, or
          • when the last flush happened more than N TUs ago

          The former would avoid smaller storefiles, the latter would put a limit on how stale an entry in the memstore can be.

          Show
          Lars Hofhansl added a comment - What should trigger the flush is an interesting discussion in itself. Should we flush: after N timeunits of write inactivity, or when the last flush happened more than N TUs ago The former would avoid smaller storefiles, the latter would put a limit on how stale an entry in the memstore can be.
          Hide
          Matt Corgan added a comment -

          Maybe add a boolean to the memstore to track if it contains edits that were not written to the WAL. No need to auto-flush in the frequent case where all edits are in the WAL.

          Show
          Matt Corgan added a comment - Maybe add a boolean to the memstore to track if it contains edits that were not written to the WAL. No need to auto-flush in the frequent case where all edits are in the WAL.
          Hide
          Jean-Daniel Cryans added a comment -

          No need to auto-flush in the frequent case where all edits are in the WAL.

          And we already roll every hour. From LogRoller:

          this.rollperiod = this.server.getConfiguration().getLong("hbase.regionserver.logroll.period", 3600000);

          Meaning that your data in the WAL can only be sitting there for so long.

          Show
          Jean-Daniel Cryans added a comment - No need to auto-flush in the frequent case where all edits are in the WAL. And we already roll every hour. From LogRoller: this.rollperiod = this.server.getConfiguration().getLong("hbase.regionserver.logroll.period", 3600000); Meaning that your data in the WAL can only be sitting there for so long.
          Hide
          Todd Lipcon added a comment -

          Maybe add a boolean to the memstore to track if it contains edits that were not written to the WAL

          HBASE-5886 adds code which tracks how much un-WAL-ed data is in the memstore.

          Meaning that your data in the WAL can only be sitting there for so long.

          But if we retain 20 or so HLogs, and we roll only every hour, then we still have 20 hours worth of data sitting there unflushed, which might be a little strange if the cluster is entirely idle.

          Show
          Todd Lipcon added a comment - Maybe add a boolean to the memstore to track if it contains edits that were not written to the WAL HBASE-5886 adds code which tracks how much un-WAL-ed data is in the memstore. Meaning that your data in the WAL can only be sitting there for so long. But if we retain 20 or so HLogs, and we roll only every hour, then we still have 20 hours worth of data sitting there unflushed, which might be a little strange if the cluster is entirely idle.
          Hide
          binlijin added a comment -

          This feature looks good.

          Show
          binlijin added a comment - This feature looks good.
          Hide
          Andrew Purtell added a comment -

          +1 We basically do the same thing as proposed but on the client side with a shared DAO layer.

          Show
          Andrew Purtell added a comment - +1 We basically do the same thing as proposed but on the client side with a shared DAO layer.
          Hide
          Matt Corgan added a comment -

          Periodically flushing the memstore seems like a good feature to me. Could also help clear out cold data from memory to make more room for bigger memstores on regions that are actually being used.

          A different solution to the underlying data loss issue might be to have a third client setting for WAL writing: NONE, SYNC, and ASYNC. ASYNC would write the data to a memory buffer, return success to the client, and another thread would flush the buffer to the WAL. The WAL would ideally only lag a few seconds behind the memstores, but some form of throttling would probably be needed.

          Show
          Matt Corgan added a comment - Periodically flushing the memstore seems like a good feature to me. Could also help clear out cold data from memory to make more room for bigger memstores on regions that are actually being used. A different solution to the underlying data loss issue might be to have a third client setting for WAL writing: NONE, SYNC, and ASYNC. ASYNC would write the data to a memory buffer, return success to the client, and another thread would flush the buffer to the WAL. The WAL would ideally only lag a few seconds behind the memstores, but some form of throttling would probably be needed.
          Hide
          stack added a comment -

          Is our deferred flush == ASYNC described above?

          Show
          stack added a comment - Is our deferred flush == ASYNC described above?
          Hide
          Lars Hofhansl added a comment -

          That (deferred flush) is what I told my colleague to use last week.
          Would be nice if the client could control this (in addition to writeToWal, we could have writeToWalAsynchronously - or something).

          A periodic memstore flush still make sense. If I get some time next week I'll come up with a patch (unless somebody else wants to take this ).

          Show
          Lars Hofhansl added a comment - That (deferred flush) is what I told my colleague to use last week. Would be nice if the client could control this (in addition to writeToWal, we could have writeToWalAsynchronously - or something). A periodic memstore flush still make sense. If I get some time next week I'll come up with a patch (unless somebody else wants to take this ).
          Hide
          stack added a comment -

          I like idea of client saying whether to put it on deferred flush queue or whether its to be flushed immediately.

          Show
          stack added a comment - I like idea of client saying whether to put it on deferred flush queue or whether its to be flushed immediately.
          Hide
          Nicolas Liochon added a comment -

          I also think that a periodic memstore flush. Even with a WAL, it's seems safer/more efficient.
          It seems that HBase had this a long long time ago:

            <property>
              <name>hbase.regionserver.optionalcacheflushinterval</name>
              <value>1800000</value>
              <description>
              Amount of time to wait since the last time a region was flushed before
              invoking an optional cache flush (An optional cache flush is a
              flush even though memcache is not at the memcache.flush.size).
              Default: 30 minutes (in miliseconds)
              </description>
            </property>
          

          It could also be linked to major compactions (before a major compaction, flush 'old' memstore)?

          Show
          Nicolas Liochon added a comment - I also think that a periodic memstore flush. Even with a WAL, it's seems safer/more efficient. It seems that HBase had this a long long time ago: <property> <name>hbase.regionserver.optionalcacheflushinterval</name> <value>1800000</value> <description> Amount of time to wait since the last time a region was flushed before invoking an optional cache flush (An optional cache flush is a flush even though memcache is not at the memcache.flush.size). Default: 30 minutes (in miliseconds) </description> </property> It could also be linked to major compactions (before a major compaction, flush 'old' memstore)?
          Lars Hofhansl made changes -
          Field Original Value New Value
          Link This issue is related to HBASE-3707 [ HBASE-3707 ]
          Hide
          Lars Hofhansl added a comment -

          I would like to pick this up again and add a flag to Mutation to indicate deferred WAL sync. If HRegion receives a batch of Mutation of which at least one is not marked as deferred the log is sync'ed. Otherwise it is deferred.
          This will mingle well later with HBASE-5954.

          Show
          Lars Hofhansl added a comment - I would like to pick this up again and add a flag to Mutation to indicate deferred WAL sync. If HRegion receives a batch of Mutation of which at least one is not marked as deferred the log is sync'ed. Otherwise it is deferred. This will mingle well later with HBASE-5954 .
          Nicolas Liochon made changes -
          Link This issue is required by HBASE-5843 [ HBASE-5843 ]
          Devaraj Das made changes -
          Assignee Devaraj Das [ devaraj ]
          Hide
          Enis Soztutar added a comment -

          Regardless of whether mutations have deferred sync, we might always want to flush periodically. We are rolling the WAL periodically, but if we do not flush, we may end up with a lof of hlogs to recover from in case of failover.

          Show
          Enis Soztutar added a comment - Regardless of whether mutations have deferred sync, we might always want to flush periodically. We are rolling the WAL periodically, but if we do not flush, we may end up with a lof of hlogs to recover from in case of failover.
          Hide
          Devaraj Das added a comment -

          Yes, Enis Soztutar, that's my plan..

          Show
          Devaraj Das added a comment - Yes, Enis Soztutar , that's my plan..
          Hide
          Devaraj Das added a comment -

          After going some back and forth, I decided to separate the issue of having the feature on asynchronous write to WAL from the periodic flush (and Enis also suggested the same in our offline discussions).

          This is a work-in-progress patch that does the periodic flush based on when was the last flush done, and whether there are any edits after the last flush in any memstore in that region. Resurrected the config hbase.regionserver.optionalcacheflushinterval. Not tested yet.

          Show
          Devaraj Das added a comment - After going some back and forth, I decided to separate the issue of having the feature on asynchronous write to WAL from the periodic flush (and Enis also suggested the same in our offline discussions). This is a work-in-progress patch that does the periodic flush based on when was the last flush done, and whether there are any edits after the last flush in any memstore in that region. Resurrected the config hbase.regionserver.optionalcacheflushinterval. Not tested yet.
          Devaraj Das made changes -
          Attachment 5930-wip.patch [ 12566224 ]
          Devaraj Das made changes -
          Fix Version/s 0.96.0 [ 12320040 ]
          Hide
          Enis Soztutar added a comment -

          I would like to pick this up again and add a flag to Mutation to indicate deferred WAL sync. If HRegion receives a batch of Mutation of which at least one is not marked as deferred the log is sync'ed. Otherwise it is deferred.

          I like the idea of having a deferred flush at the Put level. Now the weird thing is that it is per table, not per column family. I guess we can have per-table/per-cf or per batch deferred flush setting.
          With this, maybe we will no longer need skipWAL if we can prove that deferred flush is as fast as skip WAL. Most of the time, we actually do not want to skip WAL, we just want a deferred flush.

          I decided to separate the issue of having the feature on asynchronous write to WAL from the periodic flush

          +1 on doing separating the two.

          Show
          Enis Soztutar added a comment - I would like to pick this up again and add a flag to Mutation to indicate deferred WAL sync. If HRegion receives a batch of Mutation of which at least one is not marked as deferred the log is sync'ed. Otherwise it is deferred. I like the idea of having a deferred flush at the Put level. Now the weird thing is that it is per table, not per column family. I guess we can have per-table/per-cf or per batch deferred flush setting. With this, maybe we will no longer need skipWAL if we can prove that deferred flush is as fast as skip WAL. Most of the time, we actually do not want to skip WAL, we just want a deferred flush. I decided to separate the issue of having the feature on asynchronous write to WAL from the periodic flush +1 on doing separating the two.
          Hide
          Lars Hofhansl added a comment -

          I'd be fine with this in 0.94 as well.

          Show
          Lars Hofhansl added a comment - I'd be fine with this in 0.94 as well.
          Hide
          Devaraj Das added a comment -

          Simplified the logic that checks whether to do flush or not (HRegion.shouldFlush()).

          Show
          Devaraj Das added a comment - Simplified the logic that checks whether to do flush or not (HRegion.shouldFlush()).
          Devaraj Das made changes -
          Attachment 5930-1.patch [ 12566262 ]
          Hide
          Nicolas Liochon added a comment -

          With this, maybe we will no longer need skipWAL if we can prove that deferred flush is as fast as skip WAL.

          In standard database, skipping the WAL is often used when you're doing a functional upgrade requiring some unavailability time, i.e.:

          • dump
          • run batch scripts to update your data
          • if anything goes wrong reload the dump

          For hundreds of reasons it makes much less sense with HBase, but it could happen (some companies don't need 24x24). So we should not remove the skipWAL imho, except if it really simplify something internally.

          On the patch itself, I have a question on adding some randomness. The scenario I'm thinking about is a massive but periodic update on a table: all the regions will be written simultaneously, hence flushed simultaneously. That's the main use case for this JIRA, and this could hammer the namenode, imho. Except if we thing there is enough randomness by having a different flusher by regionserver (which may not be the case if all regions servers are started simultaneously).

          As a side note, I would personally like a flush interval of 10 minutes:

          • it would help on .META. recovery, especially with the separate wal for .META.
          • this allows to have more regions: today, on average and in theory, each region takes 50% of an hdfs block size of memory. The more regions we flush early, the more empty memstore we have...
          Show
          Nicolas Liochon added a comment - With this, maybe we will no longer need skipWAL if we can prove that deferred flush is as fast as skip WAL. In standard database, skipping the WAL is often used when you're doing a functional upgrade requiring some unavailability time, i.e.: dump run batch scripts to update your data if anything goes wrong reload the dump For hundreds of reasons it makes much less sense with HBase, but it could happen (some companies don't need 24x24). So we should not remove the skipWAL imho, except if it really simplify something internally. On the patch itself, I have a question on adding some randomness. The scenario I'm thinking about is a massive but periodic update on a table: all the regions will be written simultaneously, hence flushed simultaneously. That's the main use case for this JIRA, and this could hammer the namenode, imho. Except if we thing there is enough randomness by having a different flusher by regionserver (which may not be the case if all regions servers are started simultaneously). As a side note, I would personally like a flush interval of 10 minutes: it would help on .META. recovery, especially with the separate wal for .META. this allows to have more regions: today, on average and in theory, each region takes 50% of an hdfs block size of memory. The more regions we flush early, the more empty memstore we have...
          Hide
          Lars Hofhansl added a comment -

          How can deferred log flush ever be as fast as not writing the WAL at all?
          Considering only the latency of a single request that might be true in many cases, but it will definitely not be true on a busy cluster since all data is written to the disks twice.

          Show
          Lars Hofhansl added a comment - How can deferred log flush ever be as fast as not writing the WAL at all? Considering only the latency of a single request that might be true in many cases, but it will definitely not be true on a busy cluster since all data is written to the disks twice.
          Hide
          Ted Yu added a comment -

          Where is PeriodicMemstoreFlusher instantiated ?
          Currently MEMSTORE_PERIODIC_FLUSH_INTERVAL is read by both HRegion and PeriodicMemstoreFlusher.

          +  boolean shouldFlush() {
          

          Can we pass the interval to the above method so that HRegion doesn't need to introduce:

          +  private long flushCheckInterval;
          

          What value for MEMSTORE_PERIODIC_FLUSH_INTERVAL would be interpreted as disabling the periodic flush ?

          Thanks

          Show
          Ted Yu added a comment - Where is PeriodicMemstoreFlusher instantiated ? Currently MEMSTORE_PERIODIC_FLUSH_INTERVAL is read by both HRegion and PeriodicMemstoreFlusher. + boolean shouldFlush() { Can we pass the interval to the above method so that HRegion doesn't need to introduce: + private long flushCheckInterval; What value for MEMSTORE_PERIODIC_FLUSH_INTERVAL would be interpreted as disabling the periodic flush ? Thanks
          Hide
          Devaraj Das added a comment -

          Lars Hofhansl, yeah what Enis Soztutar meant IMHO is that the latency from the client's point of view would improve when deferred flush is used for the mutations. Also, we considered the case that users would most likely not want to skip WAL if we promise them that there wouldn't be latency issues (maybe on a busy cluster). But yeah, it'd not make a difference on the overall IOPS in the cluster...

          Nicolas Liochon, generally agree with you that we should not remove the skipWal option without giving it a real good thought and before considering more use cases. And, yes the idea of randomizing the flushes across regionservers sounds good. I'll think up how to incorporate that.

          Ted Yu, good catch on the instantiation I was focusing on getting the logic right; forgot to instantiate the chore. I'd prefer to leave the shouldFlush() signature as is (it's a matter of implementation that the shouldFlush method implementation is using the same constant underneath but it could be very well a different constant or shouldFlush implementation could be different sometime when this constant is not even used..).

          Show
          Devaraj Das added a comment - Lars Hofhansl , yeah what Enis Soztutar meant IMHO is that the latency from the client's point of view would improve when deferred flush is used for the mutations. Also, we considered the case that users would most likely not want to skip WAL if we promise them that there wouldn't be latency issues (maybe on a busy cluster). But yeah, it'd not make a difference on the overall IOPS in the cluster... Nicolas Liochon , generally agree with you that we should not remove the skipWal option without giving it a real good thought and before considering more use cases. And, yes the idea of randomizing the flushes across regionservers sounds good. I'll think up how to incorporate that. Ted Yu , good catch on the instantiation I was focusing on getting the logic right; forgot to instantiate the chore. I'd prefer to leave the shouldFlush() signature as is (it's a matter of implementation that the shouldFlush method implementation is using the same constant underneath but it could be very well a different constant or shouldFlush implementation could be different sometime when this constant is not even used..).
          Hide
          Ted Yu added a comment -

          w.r.t. randomizing the flushes across regionservers, one approach is to introduce a new znode whose data is the outstanding count of flush requests, cluster wise. We place an upper bound on this count. PeriodicMemstoreFlusher wouldn't create new request if the count is at upper bound.

          Show
          Ted Yu added a comment - w.r.t. randomizing the flushes across regionservers, one approach is to introduce a new znode whose data is the outstanding count of flush requests, cluster wise. We place an upper bound on this count. PeriodicMemstoreFlusher wouldn't create new request if the count is at upper bound.
          Hide
          Lars Hofhansl added a comment -

          That would work (using a znode). I do think it's fine to place an upper limit per regionserver, and maybe we won't need an upper limit at all.
          I so like the idea of some randomness. We could stagger per memstore and add a random jigger that could be up to 1/2 (just making this up, though) of the flush interval. We'd get a new random jigger number after each flush and at memstore creation.

          Show
          Lars Hofhansl added a comment - That would work (using a znode). I do think it's fine to place an upper limit per regionserver, and maybe we won't need an upper limit at all. I so like the idea of some randomness. We could stagger per memstore and add a random jigger that could be up to 1/2 (just making this up, though) of the flush interval. We'd get a new random jigger number after each flush and at memstore creation.
          Hide
          Devaraj Das added a comment -

          In this patch, I added a random sleep (upto the two minutes) before flushes. Changed the default flush interval to 10 minutes.

          Show
          Devaraj Das added a comment - In this patch, I added a random sleep (upto the two minutes) before flushes. Changed the default flush interval to 10 minutes.
          Devaraj Das made changes -
          Attachment 5930-2.1.patch [ 12566533 ]
          Hide
          Devaraj Das added a comment -

          Trying hudson

          Show
          Devaraj Das added a comment - Trying hudson
          Devaraj Das made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12566533/5930-2.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.TestHeapSize

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566533/5930-2.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.TestHeapSize Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4182//console This message is automatically generated.
          Hide
          Ted Yu added a comment -
          +  private Random rand = new Random();
          

          Please use SecureRandom.

          +          } catch (InterruptedException ie){
          +            //ignore
          

          Please restore interrupt status.

          Should upper bound for the sleep take length of MemStoreFlusher.flushQueue into consideration ?
          When many FlushQueueEntry's pile up in flushQueue, we may want to wait longer.

          Also, the sleep should be bounded by the remaining time w.r.t. cacheFlushInterval - we don't want the loop in chore() to outlast cacheFlushInterval.

          Show
          Ted Yu added a comment - + private Random rand = new Random(); Please use SecureRandom. + } catch (InterruptedException ie){ + //ignore Please restore interrupt status. Should upper bound for the sleep take length of MemStoreFlusher.flushQueue into consideration ? When many FlushQueueEntry's pile up in flushQueue, we may want to wait longer. Also, the sleep should be bounded by the remaining time w.r.t. cacheFlushInterval - we don't want the loop in chore() to outlast cacheFlushInterval.
          Hide
          Lars Hofhansl added a comment -

          Absolutely not use SecureRandom here. We're not using this to generate cryptographics keys, but just some jitter for memstore flush timing, right?
          SecureRandom will exhaust your locally generated entropy that is much better used in case where it is actually needed (and it can hang - on Linux at least - if not enough entropy has been collected)

          Show
          Lars Hofhansl added a comment - Absolutely not use SecureRandom here. We're not using this to generate cryptographics keys, but just some jitter for memstore flush timing, right? SecureRandom will exhaust your locally generated entropy that is much better used in case where it is actually needed (and it can hang - on Linux at least - if not enough entropy has been collected)
          Hide
          Lars Hofhansl added a comment -

          I think the delay should algorithmically related to the flush interval (like interval / 3 or something, could make the jitter factor configurable).
          Could we fold the jitter into shouldFlush() rather than actually waiting in chore()?

          Show
          Lars Hofhansl added a comment - I think the delay should algorithmically related to the flush interval (like interval / 3 or something, could make the jitter factor configurable). Could we fold the jitter into shouldFlush() rather than actually waiting in chore()?
          Hide
          Devaraj Das added a comment -

          I think the delay should algorithmically related to the flush interval

          I think what I currently have has a certain advantage - like if the configured value of cacheflushinterval is too low or something, the chore will be triggered very often but the sleep interval (0 - 2 minutes) would keep the #flushes under control. But yeah, I can always enforce a minimum delay before each flush.

          Could we fold the jitter into shouldFlush() rather than actually waiting in chore()?

          I think that shouldFlush shouldn't be involved in determining how much to delay the flush. Do you see any issues in waiting in the chore?

          Show
          Devaraj Das added a comment - I think the delay should algorithmically related to the flush interval I think what I currently have has a certain advantage - like if the configured value of cacheflushinterval is too low or something, the chore will be triggered very often but the sleep interval (0 - 2 minutes) would keep the #flushes under control. But yeah, I can always enforce a minimum delay before each flush. Could we fold the jitter into shouldFlush() rather than actually waiting in chore()? I think that shouldFlush shouldn't be involved in determining how much to delay the flush. Do you see any issues in waiting in the chore?
          Hide
          Devaraj Das added a comment -

          Should upper bound for the sleep take length of MemStoreFlusher.flushQueue into consideration ?

          Ted Yu, I think we don't have to worry about this one as much. The reason being that there is a random delay before each flush is inserted in the queue (as opposed to inserts coming in at a rate faster than what the flusher can handle).

          Also, the sleep should be bounded by the remaining time w.r.t. cacheFlushInterval - we don't want the loop in chore() to outlast cacheFlushInterval.

          This should be fine. I don't see issues with this one.

          Show
          Devaraj Das added a comment - Should upper bound for the sleep take length of MemStoreFlusher.flushQueue into consideration ? Ted Yu , I think we don't have to worry about this one as much. The reason being that there is a random delay before each flush is inserted in the queue (as opposed to inserts coming in at a rate faster than what the flusher can handle). Also, the sleep should be bounded by the remaining time w.r.t. cacheFlushInterval - we don't want the loop in chore() to outlast cacheFlushInterval. This should be fine. I don't see issues with this one.
          Hide
          Lars Hofhansl added a comment -

          Hmm... This is a bit more difficult than I thought.

          I think what we want to limit is this: The maximum time an unflushed edit will remain in the memstore. Otherwise one could trickle in edit 1 every hour and get very old data in the memstore.
          (Doing that could potentially also be cheaper as we do not need to retrieve the current time on each edit, just the first one after a flush).

          If that is true, then what we want track is not the time of the newest edit, but the time of oldest unflushed edit, and flush if that gets too old.
          In order to avoid flushing all memstores at the same time, we want to offset the memstores flush times.
          We can do it the way you have it.
          (but it seems natural to me to do that at the place where we detect that the memstore needs to be flushed. For this to work the chore needs to wake up more frequently than the flush interval.)

          Btw. the flush interval you have a 10mins, not 1h.

          Show
          Lars Hofhansl added a comment - Hmm... This is a bit more difficult than I thought. I think what we want to limit is this: The maximum time an unflushed edit will remain in the memstore. Otherwise one could trickle in edit 1 every hour and get very old data in the memstore. (Doing that could potentially also be cheaper as we do not need to retrieve the current time on each edit, just the first one after a flush). If that is true, then what we want track is not the time of the newest edit, but the time of oldest unflushed edit, and flush if that gets too old. In order to avoid flushing all memstores at the same time, we want to offset the memstores flush times. We can do it the way you have it. (but it seems natural to me to do that at the place where we detect that the memstore needs to be flushed. For this to work the chore needs to wake up more frequently than the flush interval.) Btw. the flush interval you have a 10mins, not 1h.
          Hide
          Devaraj Das added a comment -

          Lars Hofhansl, I think the patch addresses the use case you mention. The shouldFlush call in HRegion first does a global check whether there was a recent flush done, and if so, returns. If not, it goes over the memstores and if there is a memstore edit that happened after the last flush (and due to the previous global check, implicitly after the timeinterval), it returns true indicating that a flush is needed now. In general, if a memstore has at least one edit after the last flush, and the last flush happened before the timeinterval (let's say configured to 60 minutes), then a flush will be triggered now. The flushes are offset by a random sleep related up to a max of 2 minutes.

          The attached patch addresses a few of Ted's comments (setting the interrupt status back in the PeriodicMemstoreFlusher chore). It also puts a minimum delay of 1 sec between flushed. It makes the default sleep to 60 minutes.

          Lars Hofhansl, would appreciate if you could take a look at this please, and let me know if I missed anything in the patch w.r.t the scenario you outlined..

          Show
          Devaraj Das added a comment - Lars Hofhansl , I think the patch addresses the use case you mention. The shouldFlush call in HRegion first does a global check whether there was a recent flush done, and if so, returns. If not, it goes over the memstores and if there is a memstore edit that happened after the last flush (and due to the previous global check, implicitly after the timeinterval), it returns true indicating that a flush is needed now. In general, if a memstore has at least one edit after the last flush, and the last flush happened before the timeinterval (let's say configured to 60 minutes), then a flush will be triggered now. The flushes are offset by a random sleep related up to a max of 2 minutes. The attached patch addresses a few of Ted's comments (setting the interrupt status back in the PeriodicMemstoreFlusher chore). It also puts a minimum delay of 1 sec between flushed. It makes the default sleep to 60 minutes. Lars Hofhansl , would appreciate if you could take a look at this please, and let me know if I missed anything in the patch w.r.t the scenario you outlined..
          Devaraj Das made changes -
          Attachment 5930-2.2.patch [ 12566881 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12566881/5930-2.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.TestHeapSize

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566881/5930-2.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.TestHeapSize Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4226//console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          Attaching another rev of the patch. Changed the policies for flushing some. Here is a summary:
          0. If the last flush happened in the last 60 minutes don't request another flush now for the region in question.
          1. For a region, a flush is requested if the following is true:
          1.1. If the region is the meta region, and if there is at least one edit after the last flush request another flush now.
          1.2. If the region is not one of the meta regions, a flush is requested,
          1.2.1. if the last edit is more than 1 minute old, or,
          1.2.2. if the region wasn't flushed in the last two flush cycles (this was Enis's suggestion)

          I also added a new API in FlushRequester to do flush requests with a delay. That is used by the chore (earlier I used to throttle the flushes by doing a sleep in the chore and that seemed a little odd).

          Thoughts?

          Show
          Devaraj Das added a comment - Attaching another rev of the patch. Changed the policies for flushing some. Here is a summary: 0. If the last flush happened in the last 60 minutes don't request another flush now for the region in question. 1. For a region, a flush is requested if the following is true: 1.1. If the region is the meta region, and if there is at least one edit after the last flush request another flush now. 1.2. If the region is not one of the meta regions, a flush is requested, 1.2.1. if the last edit is more than 1 minute old, or, 1.2.2. if the region wasn't flushed in the last two flush cycles (this was Enis's suggestion) I also added a new API in FlushRequester to do flush requests with a delay. That is used by the chore (earlier I used to throttle the flushes by doing a sleep in the chore and that seemed a little odd). Thoughts?
          Devaraj Das made changes -
          Attachment 5930-2.3.patch [ 12568108 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12568108/5930-2.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestAdmin

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12568108/5930-2.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4344//console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          I should add that making the checks as described before prevents some potentially unneeded flushes, while bounding the max duration an edit lives in the memstore...

          Lars Hofhansl, could you please take a look at the patch.

          Show
          Devaraj Das added a comment - I should add that making the checks as described before prevents some potentially unneeded flushes, while bounding the max duration an edit lives in the memstore... Lars Hofhansl , could you please take a look at the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12568108/5930-2.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.security.access.TestAccessControlFilter

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12568108/5930-2.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.security.access.TestAccessControlFilter Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4369//console This message is automatically generated.
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Hide
          Nicolas Liochon added a comment -

          This one has been forgotten. Lars Hofhansl, do you have any opinion on the patch? Devaraj Das, I can finish the work (if any ) if you're busy.

          Show
          Nicolas Liochon added a comment - This one has been forgotten. Lars Hofhansl , do you have any opinion on the patch? Devaraj Das , I can finish the work (if any ) if you're busy.
          Hide
          Devaraj Das added a comment -

          Hey Nicolas Liochon, thanks for bringing this up. I was actually waiting for Lars Hofhansl to get back with his opinion on the latest patch...

          Show
          Devaraj Das added a comment - Hey Nicolas Liochon , thanks for bringing this up. I was actually waiting for Lars Hofhansl to get back with his opinion on the latest patch...
          Hide
          stack added a comment -

          Moving to 0.95.1 Has question for you Lars Hofhansl

          Show
          stack added a comment - Moving to 0.95.1 Has question for you Lars Hofhansl
          stack made changes -
          Fix Version/s 0.95.1 [ 12324288 ]
          Fix Version/s 0.95.0 [ 12324094 ]
          Hide
          stack added a comment -

          Lars Hofhansl ping. Question for you in above.

          Show
          stack added a comment - Lars Hofhansl ping. Question for you in above.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12568108/5930-2.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5418//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12568108/5930-2.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5418//console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          Updated patch w.r.t the current trunk.

          Show
          Devaraj Das added a comment - Updated patch w.r.t the current trunk.
          Devaraj Das made changes -
          Attachment 5930-2.4.patch [ 12580196 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580196/5930-2.4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580196/5930-2.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5422//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          I'd have to reread the patch.
          The semantics that we should achieve is a maximum time for an unlogged KV to remain in the memstore (this is different from periodic flushing - I misnamed this issue).

          Show
          Lars Hofhansl added a comment - I'd have to reread the patch. The semantics that we should achieve is a maximum time for an unlogged KV to remain in the memstore (this is different from periodic flushing - I misnamed this issue).
          Hide
          Devaraj Das added a comment -

          Yes, Lars, I think the patch does what you talk about.

          Show
          Devaraj Das added a comment - Yes, Lars, I think the patch does what you talk about.
          Hide
          Lars Hofhansl added a comment - - edited

          Sorry if I seem difficult with this one... How about a theme like this:

          • We record the time of the first edit made to the memstore after a flush. We can even improve this and only record the time of the first unlogged edit made.
          • Periodically we run the chore, if the recorded time of that first edit is older than a configurable X then we flush the memstore.

          That would:

          1. be simpler
          2. clearly limit the maximum an edit will stay in the memstore without being flushed
          Show
          Lars Hofhansl added a comment - - edited Sorry if I seem difficult with this one... How about a theme like this: We record the time of the first edit made to the memstore after a flush. We can even improve this and only record the time of the first unlogged edit made. Periodically we run the chore, if the recorded time of that first edit is older than a configurable X then we flush the memstore. That would: be simpler clearly limit the maximum an edit will stay in the memstore without being flushed
          Hide
          Devaraj Das added a comment -

          Lars, I'd appreciate if you kindly take a look at the patch. The patch as it stands is simple and also limits the maximum time an edit will stay un-flushed. The high level outline is here https://issues.apache.org/jira/browse/HBASE-5930?focusedCommentId=13571813&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13571813 (I have also put detailed comments in the shouldFlush method added in the patch)

          Show
          Devaraj Das added a comment - Lars, I'd appreciate if you kindly take a look at the patch. The patch as it stands is simple and also limits the maximum time an edit will stay un-flushed. The high level outline is here https://issues.apache.org/jira/browse/HBASE-5930?focusedCommentId=13571813&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13571813 (I have also put detailed comments in the shouldFlush method added in the patch)
          Hide
          Lars Hofhansl added a comment -

          Why is the approach in the patch better than what I have described?

          I believe the approach I described is better in the following ways:

          • The logic is simpler
          • We directly measure the age of oldest edit in the memstore, which is the exact metric we want to limit
          • We only have track the current time for the first KV inserted into the memstore after a flush (System.currentTimeMillis() is not free)

          I'm happy to make a sample patch, then we can decide on the merit of the two patches.

          Show
          Lars Hofhansl added a comment - Why is the approach in the patch better than what I have described? I believe the approach I described is better in the following ways: The logic is simpler We directly measure the age of oldest edit in the memstore, which is the exact metric we want to limit We only have track the current time for the first KV inserted into the memstore after a flush (System.currentTimeMillis() is not free) I'm happy to make a sample patch, then we can decide on the merit of the two patches.
          Hide
          Lars Hofhansl added a comment -

          Here's a sample patch of what I mean.

          Show
          Lars Hofhansl added a comment - Here's a sample patch of what I mean.
          Lars Hofhansl made changes -
          Attachment 5930-track-oldest-sample.txt [ 12580468 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580468/5930-track-oldest-sample.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580468/5930-track-oldest-sample.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5444//console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          Cool. This looks like it exactly does what you stated in your last comment. +1 (not sure why releaseaudit failed, even though the structure of the patch is essentially the same as what I submitted before). I'd like to commit this tomorrow morning unless I hear otherwise from anyone.

          Show
          Devaraj Das added a comment - Cool. This looks like it exactly does what you stated in your last comment. +1 (not sure why releaseaudit failed, even though the structure of the patch is essentially the same as what I submitted before). I'd like to commit this tomorrow morning unless I hear otherwise from anyone.
          Hide
          ramkrishna.s.vasudevan added a comment -

          +1 on the Lars patch.

          Show
          ramkrishna.s.vasudevan added a comment - +1 on the Lars patch.
          Hide
          Devaraj Das added a comment -

          Don't think the releaseaudit warning is caused by the last patch. It seems to be there in the other builds prior to this pre-commit build as well (is HBASE-8431 the fix?). I ran the releaseaudit test locally and it passed.

          Show
          Devaraj Das added a comment - Don't think the releaseaudit warning is caused by the last patch. It seems to be there in the other builds prior to this pre-commit build as well (is HBASE-8431 the fix?). I ran the releaseaudit test locally and it passed.
          Hide
          Devaraj Das added a comment -

          I committed this. Thanks for the last update on the patch, Lars.

          Show
          Devaraj Das added a comment - I committed this. Thanks for the last update on the patch, Lars.
          Devaraj Das made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Lars Hofhansl added a comment -

          Whoa. You guys are fast. This was more of a sample patch
          I'll do a bit more double checking today.

          I would also like to have this 0.94. Any objections to that?

          Show
          Lars Hofhansl added a comment - Whoa. You guys are fast. This was more of a sample patch I'll do a bit more double checking today. I would also like to have this 0.94. Any objections to that?
          Lars Hofhansl made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          Hide
          Devaraj Das added a comment -

          Sure Lars, do the due diligence from your side. No objections for commit to 0.94.

          Show
          Devaraj Das added a comment - Sure Lars, do the due diligence from your side. No objections for commit to 0.94.
          Lars Hofhansl made changes -
          Summary Periodically flush the Memstore? Limits the amount of time an edit can live in the memstore.
          Hide
          Lars Hofhansl added a comment -

          Let me keep that open until I have the 0.94 patch.

          Show
          Lars Hofhansl added a comment - Let me keep that open until I have the 0.94 patch.
          Lars Hofhansl made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.8 [ 12324145 ]
          Hide
          Lars Hofhansl added a comment -

          And we should have had a test too.

          Can you think something up DD?

          Show
          Lars Hofhansl added a comment - And we should have had a test too. Can you think something up DD?
          Hide
          Lars Hofhansl added a comment -

          Here's a 0.94 patch. Please have a look.

          Show
          Lars Hofhansl added a comment - Here's a 0.94 patch. Please have a look.
          Lars Hofhansl made changes -
          Attachment 5930-0.94.txt [ 12580577 ]
          Hide
          Devaraj Das added a comment -

          Okay.. Will think of a test. Will look at the 0.94 patch.

          Show
          Devaraj Das added a comment - Okay.. Will think of a test. Will look at the 0.94 patch.
          Hide
          Elliott Clark added a comment -

          hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java got the header removed with this commit. Here's a quick addendum.

          Show
          Elliott Clark added a comment - hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java got the header removed with this commit. Here's a quick addendum.
          Elliott Clark made changes -
          Attachment HBASE-5930-ADD-0.patch [ 12580587 ]
          Hide
          Devaraj Das added a comment -

          Thanks, Elliott for catching this. I was testing releaseaudit - randomly chose a java file and removed the header with an intent to make releaseaudit fail. Forgot to revert the change before my commit.

          Show
          Devaraj Das added a comment - Thanks, Elliott for catching this. I was testing releaseaudit - randomly chose a java file and removed the header with an intent to make releaseaudit fail. Forgot to revert the change before my commit.
          Hide
          Elliott Clark added a comment -

          Comitted the addendum and rat is clean again. Thanks Devaraj Das

          Show
          Elliott Clark added a comment - Comitted the addendum and rat is clean again. Thanks Devaraj Das
          Hide
          Enis Soztutar added a comment -

          This is causing the test at patch HBASE-2231 to fail. The reason is that we had a residual configuration in hbase-site.xml from earlier:

           <name>hbase.regionserver.optionalcacheflushinterval</name>
           <value>1000</value>
          

          I also was not able to understand when we put a delayed flush request to the queue, and later another request with no delay comes, it seems we will just ignore the latter one. Shouldn't we honor the no-delay one before waiting on the delay?

          Show
          Enis Soztutar added a comment - This is causing the test at patch HBASE-2231 to fail. The reason is that we had a residual configuration in hbase-site.xml from earlier: <name>hbase.regionserver.optionalcacheflushinterval</name> <value>1000</value> I also was not able to understand when we put a delayed flush request to the queue, and later another request with no delay comes, it seems we will just ignore the latter one. Shouldn't we honor the no-delay one before waiting on the delay?
          Enis Soztutar made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Hide
          Enis Soztutar added a comment -

          Attaching addendum2 patch. Also the logs from the test execution.

          Show
          Enis Soztutar added a comment - Attaching addendum2 patch. Also the logs from the test execution.
          Enis Soztutar made changes -
          Attachment hbase-5930-addendum2.patch [ 12580601 ]
          Attachment hbase-5930-test-execution.log [ 12580602 ]
          Hide
          Lars Hofhansl added a comment -

          2nd addendum is good. Will add that to the 0.94 patch as well.
          Are we good with the patch as is, or should we revert and work through the details?

          Show
          Lars Hofhansl added a comment - 2nd addendum is good. Will add that to the 0.94 patch as well. Are we good with the patch as is, or should we revert and work through the details?
          Hide
          Devaraj Das added a comment -

          I also was not able to understand when we put a delayed flush request to the queue, and later another request with no delay comes, it seems we will just ignore the latter one. Shouldn't we honor the no-delay one before waiting on the delay?

          Makes sense to honor the no-delay one. I'll open an issue to take this into account.

          Show
          Devaraj Das added a comment - I also was not able to understand when we put a delayed flush request to the queue, and later another request with no delay comes, it seems we will just ignore the latter one. Shouldn't we honor the no-delay one before waiting on the delay? Makes sense to honor the no-delay one. I'll open an issue to take this into account.
          Hide
          Enis Soztutar added a comment -

          Are we good with the patch as is, or should we revert and work through the details?

          Either that or new issue. Whichever is the cleanest.

          Show
          Enis Soztutar added a comment - Are we good with the patch as is, or should we revert and work through the details? Either that or new issue. Whichever is the cleanest.
          Hide
          Devaraj Das added a comment -

          Lars, I think the addendums fixes the issues to do with rat and the test, but I'd like to open a separate issue on the queuing of flush requests. I'll get a patch by today (worst case tomorrow) and will do a 0.94 patch for that as well..

          Show
          Devaraj Das added a comment - Lars, I think the addendums fixes the issues to do with rat and the test, but I'd like to open a separate issue on the queuing of flush requests. I'll get a patch by today (worst case tomorrow) and will do a 0.94 patch for that as well..
          Hide
          Lars Hofhansl added a comment -

          Sounds good. We have time until 0.94.8, so we're good, unless stack want's to release 0.95.1 tomorrow.

          Show
          Lars Hofhansl added a comment - Sounds good. We have time until 0.94.8, so we're good, unless stack want's to release 0.95.1 tomorrow.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #4080 (See https://builds.apache.org/job/HBase-TRUNK/4080/)
          HBASE-5930. Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475990)
          HBASE-5930 Limits the amount of time an edit can live in the memstore. (Revision 1475970)
          HBASE-5930. Limits the amount of time an edit can live in the memstore. (Revision 1475872)

          Result = FAILURE
          ddas :
          Files :

          • /hbase/trunk/hbase-server/src/test/resources/hbase-site.xml

          eclark :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java

          ddas :
          Files :

          • /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #4080 (See https://builds.apache.org/job/HBase-TRUNK/4080/ ) HBASE-5930 . Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475990) HBASE-5930 Limits the amount of time an edit can live in the memstore. (Revision 1475970) HBASE-5930 . Limits the amount of time an edit can live in the memstore. (Revision 1475872) Result = FAILURE ddas : Files : /hbase/trunk/hbase-server/src/test/resources/hbase-site.xml eclark : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java ddas : Files : /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          Hide
          Devaraj Das added a comment -

          I'd like to open a separate issue on the queuing of flush requests

          After some deliberations and code browsing, it seems like we don't need to do this. The reasons being:

          1. We already have the emergency flush procedure in place (MemStoreFlusher.flushRegion with an emergencyFlush boolean argument exists that is called when there is an urgent need to flush).

          2. In the normal steady state of affairs, entries from the queue are processed asynchronously anyway. Flushes are done by a thread and there is no guarantee when the thread picks a flush request up. By adding a delay, this situation is not worsened much, and especially in combination with (1) above.

          3. To address the issue raised, I was thinking that when a new flush request comes without a delay, we could remove an existing delayed-flush-request entry that was queued before, and queue a new one without a delay. However, there is a complication here - there is logic that queues up entries with delayed flushes in MemStoreFlusher.flushRegion(final FlushRegionEntry). That would be impacted if I did what I said.

          All in all, this doesn't seem like an issue.. Enis Soztutar also ack'ed this in our offline discussions.

          Show
          Devaraj Das added a comment - I'd like to open a separate issue on the queuing of flush requests After some deliberations and code browsing, it seems like we don't need to do this. The reasons being: 1. We already have the emergency flush procedure in place (MemStoreFlusher.flushRegion with an emergencyFlush boolean argument exists that is called when there is an urgent need to flush). 2. In the normal steady state of affairs, entries from the queue are processed asynchronously anyway. Flushes are done by a thread and there is no guarantee when the thread picks a flush request up. By adding a delay, this situation is not worsened much, and especially in combination with (1) above. 3. To address the issue raised, I was thinking that when a new flush request comes without a delay, we could remove an existing delayed-flush-request entry that was queued before, and queue a new one without a delay. However, there is a complication here - there is logic that queues up entries with delayed flushes in MemStoreFlusher.flushRegion(final FlushRegionEntry) . That would be impacted if I did what I said. All in all, this doesn't seem like an issue.. Enis Soztutar also ack'ed this in our offline discussions.
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #163 (See https://builds.apache.org/job/hbase-0.95/163/)
          HBASE-5930. Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475991)
          HBASE-5930. Limits the amount of time an edit can live in the memstore. (Revision 1475874)

          Result = FAILURE
          ddas :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/resources/hbase-site.xml

          ddas :
          Files :

          • /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #163 (See https://builds.apache.org/job/hbase-0.95/163/ ) HBASE-5930 . Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475991) HBASE-5930 . Limits the amount of time an edit can live in the memstore. (Revision 1475874) Result = FAILURE ddas : Files : /hbase/branches/0.95/hbase-server/src/test/resources/hbase-site.xml ddas : Files : /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #81 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/81/)
          HBASE-5930. Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475991)
          HBASE-5930. Limits the amount of time an edit can live in the memstore. (Revision 1475874)

          Result = FAILURE
          ddas :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/resources/hbase-site.xml

          ddas :
          Files :

          • /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #81 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/81/ ) HBASE-5930 . Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475991) HBASE-5930 . Limits the amount of time an edit can live in the memstore. (Revision 1475874) Result = FAILURE ddas : Files : /hbase/branches/0.95/hbase-server/src/test/resources/hbase-site.xml ddas : Files : /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #511 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/511/)
          HBASE-5930. Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475990)
          HBASE-5930 Limits the amount of time an edit can live in the memstore. (Revision 1475970)
          HBASE-5930. Limits the amount of time an edit can live in the memstore. (Revision 1475872)

          Result = FAILURE
          ddas :
          Files :

          • /hbase/trunk/hbase-server/src/test/resources/hbase-site.xml

          eclark :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java

          ddas :
          Files :

          • /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #511 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/511/ ) HBASE-5930 . Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475990) HBASE-5930 Limits the amount of time an edit can live in the memstore. (Revision 1475970) HBASE-5930 . Limits the amount of time an edit can live in the memstore. (Revision 1475872) Result = FAILURE ddas : Files : /hbase/trunk/hbase-server/src/test/resources/hbase-site.xml eclark : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java ddas : Files : /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          Hide
          Lars Hofhansl added a comment -

          Should I close this one and open a 0.94 backport patch?

          Show
          Lars Hofhansl added a comment - Should I close this one and open a 0.94 backport patch?
          Hide
          Devaraj Das added a comment -

          Sorry for the delay in getting back with a review on the 0.94 patch. I will get to it tomorrow. I am fine with a new jira for the 0.94 patch or using this one itself...

          Show
          Devaraj Das added a comment - Sorry for the delay in getting back with a review on the 0.94 patch. I will get to it tomorrow. I am fine with a new jira for the 0.94 patch or using this one itself...
          Hide
          Devaraj Das added a comment -

          Lars's patch with the addendum that Enis had submitted.

          Show
          Devaraj Das added a comment - Lars's patch with the addendum that Enis had submitted.
          Devaraj Das made changes -
          Attachment 5930-0.94-added-addendum.txt [ 12581723 ]
          Hide
          Lars Hofhansl added a comment -

          Cool. If there are no objections I will commit this to 0.94 as well.
          (Should this be disabled by default in 0.94? I can see this both ways.)

          Show
          Lars Hofhansl added a comment - Cool. If there are no objections I will commit this to 0.94 as well. (Should this be disabled by default in 0.94? I can see this both ways.)
          Hide
          Enis Soztutar added a comment -

          +1 for this going to 0.94.

          Show
          Enis Soztutar added a comment - +1 for this going to 0.94.
          Hide
          Jean-Daniel Cryans added a comment -

          Would also be good to have a release note here since it's a new feature coming along with new configs.

          Show
          Jean-Daniel Cryans added a comment - Would also be good to have a release note here since it's a new feature coming along with new configs.
          Lars Hofhansl made changes -
          Release Note This feature limits the time that unflushed data will stay in the memstore.
          By default a memstore will flush if data older than 1h (3600000ms) is present.

          This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 3600000).
          Hide
          Lars Hofhansl added a comment -

          Added release notes. I also would like add an option to disable this feature (for example by setting base.regionserver.optionalcacheflushinterval to a value <= 0).

          Show
          Lars Hofhansl added a comment - Added release notes. I also would like add an option to disable this feature (for example by setting base.regionserver.optionalcacheflushinterval to a value <= 0).
          Hide
          Devaraj Das added a comment -

          Patch for trunk with the config for disabling this behavior.

          Complete patch for 0.94.

          Show
          Devaraj Das added a comment - Patch for trunk with the config for disabling this behavior. Complete patch for 0.94.
          Devaraj Das made changes -
          Attachment 5930-addendum-for-disabling.trunk.txt [ 12582788 ]
          Attachment 5930-0.94-2.txt [ 12582789 ]
          Hide
          Lars Hofhansl added a comment -

          Thanks DD (you beat me to the patch ). Was more thinking about not starting the chore at at all when disabled.
          Then again, we do have per table (or CF) configs now, right? In that case this approach would be the right one.

          Show
          Lars Hofhansl added a comment - Thanks DD (you beat me to the patch ). Was more thinking about not starting the chore at at all when disabled. Then again, we do have per table (or CF) configs now, right? In that case this approach would be the right one.
          Hide
          Devaraj Das added a comment -

          Lars Hofhansl, yes, the configs could be per table/CF. All right, this patch has some unit tests as you earlier asked for it. If you are okay with it, I'll commit it in trunk/0.95, and post an updated one for 0.94.

          Show
          Devaraj Das added a comment - Lars Hofhansl , yes, the configs could be per table/CF. All right, this patch has some unit tests as you earlier asked for it. If you are okay with it, I'll commit it in trunk/0.95, and post an updated one for 0.94.
          Devaraj Das made changes -
          Hide
          Devaraj Das added a comment -

          Had attached an incorrect patch earlier. This is the right one.

          Show
          Devaraj Das added a comment - Had attached an incorrect patch earlier. This is the right one.
          Devaraj Das made changes -
          Hide
          Lars Hofhansl added a comment -

          Nice. +1 on addendum.

          Show
          Lars Hofhansl added a comment - Nice. +1 on addendum.
          Hide
          Devaraj Das added a comment -

          Patch for 0.94 with the latest config/tests changes.

          Show
          Devaraj Das added a comment - Patch for 0.94 with the latest config/tests changes.
          Devaraj Das made changes -
          Attachment 5930-0.94-added-addendum-with-tests.txt [ 12583054 ]
          Hide
          Lars Hofhansl added a comment -

          This is the one for 0.94
          Will commit tomorrow unless I hear objections.

          Show
          Lars Hofhansl added a comment - This is the one for 0.94 Will commit tomorrow unless I hear objections.
          Hide
          Lars Hofhansl added a comment -

          Committed addendum to 0.95 and full patch to 0.94.
          DD committed addendum to trunk on Monday. Closing.

          Show
          Lars Hofhansl added a comment - Committed addendum to 0.95 and full patch to 0.94. DD committed addendum to trunk on Monday. Closing.
          Lars Hofhansl made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #141 (See https://builds.apache.org/job/HBase-0.94-security/141/)
          HBASE-5930 Limits the amount of time an edit can live in the memstore. (Davaraj and LarsH) (Revision 1483022)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/security/src/test/resources/hbase-site.xml
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.94/src/main/resources/hbase-default.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • /hbase/branches/0.94/src/test/resources/hbase-site.xml
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #141 (See https://builds.apache.org/job/HBase-0.94-security/141/ ) HBASE-5930 Limits the amount of time an edit can live in the memstore. (Davaraj and LarsH) (Revision 1483022) Result = FAILURE larsh : Files : /hbase/branches/0.94/security/src/test/resources/hbase-site.xml /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/resources/hbase-default.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java /hbase/branches/0.94/src/test/resources/hbase-site.xml
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #196 (See https://builds.apache.org/job/hbase-0.95/196/)
          HBASE-5930 Addendum with tests. (Davaraj) (Revision 1483037)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #196 (See https://builds.apache.org/job/hbase-0.95/196/ ) HBASE-5930 Addendum with tests. (Davaraj) (Revision 1483037) Result = FAILURE larsh : Files : /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #984 (See https://builds.apache.org/job/HBase-0.94/984/)
          HBASE-5930 Limits the amount of time an edit can live in the memstore. (Davaraj and LarsH) (Revision 1483022)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/security/src/test/resources/hbase-site.xml
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.94/src/main/resources/hbase-default.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • /hbase/branches/0.94/src/test/resources/hbase-site.xml
          Show
          Hudson added a comment - Integrated in HBase-0.94 #984 (See https://builds.apache.org/job/HBase-0.94/984/ ) HBASE-5930 Limits the amount of time an edit can live in the memstore. (Davaraj and LarsH) (Revision 1483022) Result = SUCCESS larsh : Files : /hbase/branches/0.94/security/src/test/resources/hbase-site.xml /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/resources/hbase-default.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java /hbase/branches/0.94/src/test/resources/hbase-site.xml
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #101 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/101/)
          HBASE-5930 Addendum with tests. (Davaraj) (Revision 1483037)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #101 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/101/ ) HBASE-5930 Addendum with tests. (Davaraj) (Revision 1483037) Result = FAILURE larsh : Files : /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
          yang ming made changes -
          Environment 11
          yang ming made changes -
          Environment 11
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Devaraj Das
              Reporter:
              Lars Hofhansl
            • Votes:
              1 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development