HBase
  1. HBase
  2. HBASE-10499

In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.98.0
    • Fix Version/s: 1.0.0, 2.0.0, 0.98.10, 1.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I got this while testing 0.98RC. But am not sure if it is specific to this version. Doesn't seem so to me.
      Also it is something similar to HBASE-5312 and HBASE-5568.

      Using 10 threads i do writes to 4 RS using YCSB. The table created has 200 regions. In one of the run with 0.98 server and 0.98 client I faced this problem like the hlogs became more and the system requested flushes for those many regions.
      One by one everything was flushed except one and that one thing remained unflushed. The ripple effect of this on the client side

      com.yahoo.ycsb.DBException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 54 actions: RegionTooBusyException: 54 times,
              at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:245)
              at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
              at com.yahoo.ycsb.ClientThread.run(Client.java:307)
      Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 54 actions: RegionTooBusyException: 54 times,
              at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:187)
              at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:171)
              at org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:897)
              at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:961)
              at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1225)
              at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:232)
              ... 2 more
      

      On one of the RS

      2014-02-11 08:45:58,714 INFO  [regionserver60020.logRoller] wal.FSHLog: Too many hlogs: logs=38, maxlogs=32; forcing flush of 23 regions(s): 97d8ae2f78910cc5ded5fbb1ddad8492, d396b8a1da05c871edcb68a15608fdf2, 01a68742a1be3a9705d574ad68fec1d7, 1250381046301e7465b6cf398759378e, 127c133f47d0419bd5ab66675aff76d4, 9f01c5d25ddc6675f750968873721253, 29c055b5690839c2fa357cd8e871741e, ca4e33e3eb0d5f8314ff9a870fc43463, acfc6ae756e193b58d956cb71ccf0aa3, 187ea304069bc2a3c825bc10a59c7e84, 0ea411edc32d5c924d04bf126fa52d1e, e2f9331fc7208b1b230a24045f3c869e, d9309ca864055eddf766a330352efc7a, 1a71bdf457288d449050141b5ff00c69, 0ba9089db28e977f86a27f90bbab9717, fdbb3242d3b673bbe4790a47bc30576f, bbadaa1f0e62d8a8650080b824187850, b1a5de30d8603bd5d9022e09c574501b, cc6a9fabe44347ed65e7c325faa72030, 313b17dbff2497f5041b57fe13fa651e, 6b788c498503ddd3e1433a4cd3fb4e39, 3d71274fe4f815882e9626e1cfa050d1, acc43e4b42c1a041078774f4f20a3ff5
      ......................................................
      2014-02-11 08:47:49,580 INFO  [regionserver60020.logRoller] wal.FSHLog: Too many hlogs: logs=53, maxlogs=32; forcing flush of 2 regions(s): fdbb3242d3b673bbe4790a47bc30576f, 6b788c498503ddd3e1433a4cd3fb4e39
      
      2014-02-11 09:42:44,237 INFO  [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 16689
      2014-02-11 09:42:44,237 INFO  [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a delay of 15868
      2014-02-11 09:42:54,238 INFO  [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 20847
      2014-02-11 09:42:54,238 INFO  [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a delay of 20099
      2014-02-11 09:43:04,238 INFO  [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 8677
      
      2014-02-11 10:31:21,020 INFO  [regionserver60020.logRoller] wal.FSHLog: Too many hlogs: logs=54, maxlogs=32; forcing flush of 1 regions(s): fdbb3242d3b673bbe4790a47bc30576f
      

      I restarted another RS and there were region movements with other regions but this region stays with the RS that has this issue. One important observation is that in HRegion.internalflushCache() we need to add a debug log here

      // If nothing to flush, return and avoid logging start/stop flush.
          if (this.memstoreSize.get() <= 0) {
            return false;
          }
      

      Because we can see that the region is requsted for a flush but it does not happen and no logs related to flush are printed in the logs. so due to some reason this memstore.size() has become 0( I assume this). The earlier bugs were also due to similar reason.

      1. 10499-0.98.txt
        4 kB
        Ted Yu
      2. 10499-1.0.txt
        3 kB
        Ted Yu
      3. 10499-v8.txt
        3 kB
        Ted Yu
      4. 10499-v7.txt
        3 kB
        Ted Yu
      5. 10499-v6.txt
        3 kB
        Ted Yu
      6. 10499-v6.txt
        3 kB
        Ted Yu
      7. HBASE-10499_v5.patch
        1 kB
        ramkrishna.s.vasudevan
      8. 10499-v4.txt
        3 kB
        Ted Yu
      9. 10499-v3.txt
        3 kB
        Ted Yu
      10. 10499-v2.txt
        2 kB
        Ted Yu
      11. compaction-queue.png
        29 kB
        Matt Kapilevich
      12. rs_576f.log
        98 kB
        Honghua Feng
      13. rs_4e39.log
        36 kB
        Honghua Feng
      14. master_576f.log
        4 kB
        Honghua Feng
      15. master_4e39.log
        11 kB
        Honghua Feng
      16. hbase-root-master-ip-10-157-0-229.zip
        1005 kB
        ramkrishna.s.vasudevan
      17. workloada_0.98.dat
        35 kB
        ramkrishna.s.vasudevan
      18. HBASE-10499.patch
        0.7 kB
        ramkrishna.s.vasudevan
      19. hbase-root-regionserver-ip-10-93-128-92.zip
        2.11 MB
        ramkrishna.s.vasudevan
      20. t2.dump
        164 kB
        ramkrishna.s.vasudevan
      21. t1.dump
        159 kB
        ramkrishna.s.vasudevan

        Activity

        Hide
        ramkrishna.s.vasudevan added a comment -

        Am not sure if this could come in 0.96 and trunk also. I feel 0.96 it is possible but with trunk (recent HLog disruptor) changes am not sure. Also may be possible in 0.94.
        I don't have any soln in hand except for adding log msgs in such a case where memstoreSize could be zero. Will check more on this.

        Show
        ramkrishna.s.vasudevan added a comment - Am not sure if this could come in 0.96 and trunk also. I feel 0.96 it is possible but with trunk (recent HLog disruptor) changes am not sure. Also may be possible in 0.94. I don't have any soln in hand except for adding log msgs in such a case where memstoreSize could be zero. Will check more on this.
        Hide
        ramkrishna.s.vasudevan added a comment -

        I have logs and thread dumps taken during this time. If needed can attach them here.

        Show
        ramkrishna.s.vasudevan added a comment - I have logs and thread dumps taken during this time. If needed can attach them here.
        Hide
        ramkrishna.s.vasudevan added a comment -

        In HBASE-5568 and HBASE-5312 there were multiple flushes on the same region and splitting of regions were happening. Here no such things happen. The region in discussion has not been flushed even once and there are no splits or compactions that has happened on it.

        Show
        ramkrishna.s.vasudevan added a comment - In HBASE-5568 and HBASE-5312 there were multiple flushes on the same region and splitting of regions were happening. Here no such things happen. The region in discussion has not been flushed even once and there are no splits or compactions that has happened on it.
        Hide
        Ted Yu added a comment -

        I have logs and thread dumps taken during this time

        Please attach them.

        Show
        Ted Yu added a comment - I have logs and thread dumps taken during this time Please attach them.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Not able to reproduce this further again.

        Show
        ramkrishna.s.vasudevan added a comment - Not able to reproduce this further again.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Adding a patch that adds a log in case of memstore size is 0.

        Show
        ramkrishna.s.vasudevan added a comment - Adding a patch that adds a log in case of memstore size is 0.
        Hide
        Andrew Purtell added a comment -

        +1 for adding logging for this case. Let's get it in, rerun the test, and see if we can reproduce this issue.

        Show
        Andrew Purtell added a comment - +1 for adding logging for this case. Let's get it in, rerun the test, and see if we can reproduce this issue.
        Hide
        ramkrishna.s.vasudevan added a comment -

        committed to 0.98 alone. Will try reproducing this.

        Show
        ramkrishna.s.vasudevan added a comment - committed to 0.98 alone. Will try reproducing this.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #155 (See https://builds.apache.org/job/HBase-0.98/155/)
        HBASE-10499In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException Adding a log alone (ramkrishna: rev 1567894)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #155 (See https://builds.apache.org/job/HBase-0.98/155/ ) HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException Adding a log alone (ramkrishna: rev 1567894) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #143 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/143/)
        HBASE-10499In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException Adding a log alone (ramkrishna: rev 1567894)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #143 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/143/ ) HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException Adding a log alone (ramkrishna: rev 1567894) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        Hide
        Honghua Feng added a comment -

        Because we can see that the region is requsted for a flush but it does not happen and no logs related to flush are printed in the logs. so due to some reason this memstore.size() has become 0( I assume this)

        Reasonable guess, but it can only justify why no flush happens, but not for RegionTooBusyException is thrown by this same region(it should be the same region, right?), actually ' memstoreSize.get() <= 0' contradicts the condition for RegionTooBusyException to be thrown. This exception is thrown here(another place is when failed to lock the region for write). It's impossible for both "memstoreSize.get() <= 0" and "this.memstoreSize.get() > this.blockingMemStoreSize" hold at the same time...

        if (this.memstoreSize.get() > this.blockingMemStoreSize) {
              requestFlush();
              throw new RegionTooBusyException("Above memstore limit, " + ...memstoreSize=" + memstoreSize.get()
        

        It would help if you can attach the detailed client side log for RegionTooBusyException stack/info? would be great if with region Info and memstoreSize

        Show
        Honghua Feng added a comment - Because we can see that the region is requsted for a flush but it does not happen and no logs related to flush are printed in the logs. so due to some reason this memstore.size() has become 0( I assume this) Reasonable guess, but it can only justify why no flush happens, but not for RegionTooBusyException is thrown by this same region(it should be the same region, right?), actually ' memstoreSize.get() <= 0' contradicts the condition for RegionTooBusyException to be thrown. This exception is thrown here(another place is when failed to lock the region for write). It's impossible for both "memstoreSize.get() <= 0" and "this.memstoreSize.get() > this.blockingMemStoreSize" hold at the same time... if ( this .memstoreSize.get() > this .blockingMemStoreSize) { requestFlush(); throw new RegionTooBusyException( "Above memstore limit, " + ...memstoreSize=" + memstoreSize.get() It would help if you can attach the detailed client side log for RegionTooBusyException stack/info? would be great if with region Info and memstoreSize
        Hide
        ramkrishna.s.vasudevan added a comment -

        Check this .. I don't have the client logs. I have only the exception that i got on the client side.
        If you see the logs attached on the server you could see that the no of hlog was greater than 32.

        Show
        ramkrishna.s.vasudevan added a comment - Check this .. I don't have the client logs. I have only the exception that i got on the client side. If you see the logs attached on the server you could see that the no of hlog was greater than 32.
        Hide
        Honghua Feng added a comment -

        Thanks for check. Yes I noticed regions are selected and forced to flush due to too many hlog files. But that can only confirm the write is really heavy. The problematic region was also triggered to flush by periodicFlusher since it contains old-enough edits, which double confirms this region had not been flushed for quite a long time.
        The question is why a flush is triggered but doesn't complete all along...

        Show
        Honghua Feng added a comment - Thanks for check. Yes I noticed regions are selected and forced to flush due to too many hlog files. But that can only confirm the write is really heavy. The problematic region was also triggered to flush by periodicFlusher since it contains old-enough edits, which double confirms this region had not been flushed for quite a long time. The question is why a flush is triggered but doesn't complete all along...
        Hide
        ramkrishna.s.vasudevan added a comment -

        That is where i doubt that the memstoreSize is getting calculated wrong. Because once a flush is requested the flusher thread would try to handle it and if it is not able to flush then it would just ignore and there are no logs in that flow if you see. Again it is a guess.
        Or is there any reason why the region is not flushed from the queue, may be not possible.

        Show
        ramkrishna.s.vasudevan added a comment - That is where i doubt that the memstoreSize is getting calculated wrong. Because once a flush is requested the flusher thread would try to handle it and if it is not able to flush then it would just ignore and there are no logs in that flow if you see. Again it is a guess. Or is there any reason why the region is not flushed from the queue, may be not possible.
        Hide
        Anoop Sam John added a comment -
        2014-02-11 08:44:32,881 DEBUG [RpcServer.handler=1,port=60020] regionserver.HRegion: Flush requested on usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f.
        

        Almost sure that this is because of flush request from batchMutate() as we reached flush size after doing the puts. This request for flush itself was not continued. No trace of this region getting started to flush even. So possibility is memstoreSize got miscalculated some how ? !!

        Show
        Anoop Sam John added a comment - 2014-02-11 08:44:32,881 DEBUG [RpcServer.handler=1,port=60020] regionserver.HRegion: Flush requested on usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. Almost sure that this is because of flush request from batchMutate() as we reached flush size after doing the puts. This request for flush itself was not continued. No trace of this region getting started to flush even. So possibility is memstoreSize got miscalculated some how ? !!
        Hide
        Honghua Feng added a comment -

        Some additional thought triggered by this jira:

        A single problematic region which fails to flush all along can result in too many hlog files since its edits scatter in almost all hlog files, so even though other regions are successfully flushed and their edits are stale in hlog files, those hlog files still can't be archived because they are 'polluted' by the edits of that single problematic region. 'Polluted' means most of the edits, say maybe 99%, have been flushed to hfiles and are stale, but still non-archiveable due to the remained 1% edits from the problematic region are not flushed to hfile and still 'effective'.

        For this case, the periodicFlusher can't help though by design it aims to eliminate 'polluated' hlog files resulted from region with sparse but steady write(its writes scatter in almost all hlog files but flush can't be triggered for this region due to its total size doesn't exceed the flush threshold) by checking if it's not flushed for a long time. The reason is the region not flushed can be a problematic region, hence can't be flushed by periodicFlusher as this jira shows, so it can't eliminate 'polluated' hlog files as desired. Too many hlog files can further have bad effect such as more frequent unnecessary flush-check(always)...

        Once the number of hlog files exceeds the threshold to trigger forceful flush, that number will remain without any chance to be lower than that threshold, no matter how frequently other regions are flushed, or whether periodicFlusher triggers flush for other regions or this problematic regions, even though eventually this problematic region refuses/blocks writes to it... This is all because the older hlog files(once exceeds the threshold number) all contain some edits from this problematic region.

        But the idea of background hlog compaction thread introduced by HBASE-9873 can help to reduce number of hlog files even there are problematic un-flushable regions: it reads original hlog files, only keeps the edits of these problematic regions(and of regions not flushed yet) and write them to the new hlog files, then archive the original hlog files, which reduces the number of hlog files. Since the problematic regions will refuse writes after some time(as this jira shows), so the eventual data size of this region left in hlog files won't increase infinitely. And such problematic regions won't have that bad ripple effect for the total system.

        Show
        Honghua Feng added a comment - Some additional thought triggered by this jira: A single problematic region which fails to flush all along can result in too many hlog files since its edits scatter in almost all hlog files, so even though other regions are successfully flushed and their edits are stale in hlog files, those hlog files still can't be archived because they are 'polluted' by the edits of that single problematic region. 'Polluted' means most of the edits, say maybe 99%, have been flushed to hfiles and are stale, but still non-archiveable due to the remained 1% edits from the problematic region are not flushed to hfile and still 'effective'. For this case, the periodicFlusher can't help though by design it aims to eliminate 'polluated' hlog files resulted from region with sparse but steady write(its writes scatter in almost all hlog files but flush can't be triggered for this region due to its total size doesn't exceed the flush threshold) by checking if it's not flushed for a long time. The reason is the region not flushed can be a problematic region, hence can't be flushed by periodicFlusher as this jira shows, so it can't eliminate 'polluated' hlog files as desired. Too many hlog files can further have bad effect such as more frequent unnecessary flush-check(always)... Once the number of hlog files exceeds the threshold to trigger forceful flush, that number will remain without any chance to be lower than that threshold, no matter how frequently other regions are flushed, or whether periodicFlusher triggers flush for other regions or this problematic regions, even though eventually this problematic region refuses/blocks writes to it... This is all because the older hlog files(once exceeds the threshold number) all contain some edits from this problematic region. But the idea of background hlog compaction thread introduced by HBASE-9873 can help to reduce number of hlog files even there are problematic un-flushable regions: it reads original hlog files, only keeps the edits of these problematic regions(and of regions not flushed yet) and write them to the new hlog files, then archive the original hlog files, which reduces the number of hlog files. Since the problematic regions will refuse writes after some time(as this jira shows), so the eventual data size of this region left in hlog files won't increase infinitely. And such problematic regions won't have that bad ripple effect for the total system.
        Hide
        Honghua Feng added a comment -

        I restarted another RS and there were region movements with other regions but this region stays with the RS that has this issue

        The problematic region can't be moved to other RS is because region need to be flushed before moving out, but the problematic region can't be successfully flushed...

        Show
        Honghua Feng added a comment - I restarted another RS and there were region movements with other regions but this region stays with the RS that has this issue The problematic region can't be moved to other RS is because region need to be flushed before moving out, but the problematic region can't be successfully flushed...
        Hide
        ramkrishna.s.vasudevan added a comment -

        The problematic region can't be moved to other RS is because region need to be flushed before moving out, but the problematic region can't be successfully flushed

        Yes that is the reason.
        I like your idea in your previous comment using HBASE-9873. May be worth pursuing that. Should be useful.

        Show
        ramkrishna.s.vasudevan added a comment - The problematic region can't be moved to other RS is because region need to be flushed before moving out, but the problematic region can't be successfully flushed Yes that is the reason. I like your idea in your previous comment using HBASE-9873 . May be worth pursuing that. Should be useful.
        Hide
        Honghua Feng added a comment -

        ramkrishna.s.vasudevan, would you help provide below information? Thanks.

        1. The (active) master log file with log from '2014-02-11 08:40:00' to '2014-02-11 10:00:00'
        2. The start time(from client side log) when client received 'RegionTooBusyException'
        3. 'hbase.hstore.flusher.count', if you did configure it
        Show
        Honghua Feng added a comment - ramkrishna.s.vasudevan , would you help provide below information? Thanks. The (active) master log file with log from '2014-02-11 08:40:00' to '2014-02-11 10:00:00' The start time(from client side log) when client received 'RegionTooBusyException' 'hbase.hstore.flusher.count', if you did configure it
        Hide
        Honghua Feng added a comment -

        'hbase.hstore.flusher.count', if you did configure it

        It can be deduced from the log that this configuration is 1 (default, not configured)

        Show
        Honghua Feng added a comment - 'hbase.hstore.flusher.count', if you did configure it It can be deduced from the log that this configuration is 1 (default, not configured)
        Hide
        Honghua Feng added a comment -

        Some progress and conclusion till now for the problematic region*fdbb3242d3b673bbe4790a47bc30576f*(lack some log/info, can deduce only by code and log at hand):

        1. internalFlushcache() is never called, so root cause won't be its memstoreSize is mis-calculated ('memstoreSize <= 0' check is within internalFlushcache)
        2. On contrary memstoreSize should be > 0, and there is flushEntry in regionsInQueue, but never pop up to be executed
        3. Why it wasn't moved to other RS is not because it can't be flushed, but because HMaster didn't choose it to move out during balance... (this needs master log to confirm)

        Below are the deduction process, a bit long...

        1. HRegion's private field 'updatesLock': its writeLock().lock() is only called in internalFlushcache(), and after below log.
          LOG.debug("Started memstore flush for " + this + ", current region memstore size " +...
          

          For the problematic region, above log never be printed, so writeLock().lock() is never called => the call of lock(updatesLock.readLock(), xxx) in write operations won't fail, this means RegionTooBusyException can't be thrown in below method

          private void lock(final Lock lock, final int multiplier)
                throws RegionTooBusyException, InterruptedIOException {
              try {
                final long waitTime = Math.min(maxBusyWaitDuration,
                    busyWaitDuration * Math.min(multiplier, maxBusyWaitMultiplier));
                if (!lock.tryLock(waitTime, TimeUnit.MILLISECONDS)) {
                  throw new RegionTooBusyException(
                      "failed to get a lock in " + waitTime + " ms. " +
                          "regionName=" + (this.getRegionInfo() == null ? "unknown" :
          
        2. RegionTooBusyException occurs in two places, so it must be thrown here:
          private void checkResources()
              throws RegionTooBusyException {
              // If catalog region, do not impose resource constraints or block updates.
              if (this.getRegionInfo().isMetaRegion()) return;
          
              if (this.memstoreSize.get() > this.blockingMemStoreSize) {
                requestFlush();
                throw new RegionTooBusyException("Above memstore limit, " +
                    "regionName=" + (this.getRegionInfo() == null ? "unknown" :
          

          this means memstoreSize > 0(also means it shouldn't <= 0 in internalFlushcache())

        3. In above code, requestFlush() is called each time before RegionTooBusyException is thrown, the requestFlush()'s implementation:
          private void requestFlush() {
              if (this.rsServices == null) {
                return;
              }
              synchronized (writestate) {
                if (this.writestate.isFlushRequested()) {
                  return;
                }
                writestate.flushRequested = true;
              }
              // Make request outside of synchronize block; HBASE-818.
              this.rsServices.getFlushRequester().requestFlush(this);
              if (LOG.isDebugEnabled()) {
                LOG.debug("Flush requested on " + this);
              }
            }
          

          requestFlush is called the same times as RegionTooBusyException is thrown, but "Flush requested on" log for this region is printed only once...so in this method, it must return before the LOG.debug(), since rsServices is not null, so it must by this.writestate.isFlushRequested() is true => writestate.flushRequested should always be true each time RegionTooBusyException is thrown

        4. "Flush requested on" log does be printed once for the problematic region, means 'flushRequested' ever be set to true. And HRegion.writestate.flushRequested is set back to false only in below code besides region initialization
          public boolean flushcache() throws IOException {
            ...
            try {
                  boolean result = internalFlushcache(status);
          
                  if (coprocessorHost != null) {
                    status.setStatus("Running post-flush coprocessor hooks");
                    coprocessorHost.postFlush();
                  }
          
                  status.markComplete("Flush successful");
                  return result;
                } finally {
                  synchronized (writestate) {
                    writestate.flushing = false;
                    this.writestate.flushRequested = false;
                    writestate.notifyAll();
                  }
                }
          }
          

          This means above 'try' block never be called, otherwise the flushRequested in finally block should be set to false, above HRegion.flushcache method is the critical path for flush task coming out of flushQueue(another places calling internalFlushcache are replay split log and when closing region) ==> internalFlushcache() is never called for flush task of this problematic region from flushQueue, this also means the root cause is not 'memstoreSize <=0' (which is checked within internalFlushcache), so adding log before this statement won't help for diagnose even it reproduces next time

        5. 'forcing flush of' / 'periodicFlusher requesting flush for' means requestFlush/requestDelayedFlush are called, they all do nothing since "if (!regionsInQueue.containsKey(r))" fail, which means regionsInQueue still contains this region
        6. another problematic region is 6b788c498503ddd3e1433a4cd3fb4e39, its sympotom is exactly the same as fdbb3242d3b673bbe4790a47bc30576f, except it's flushed at '2014-02-11 09:51:11' due to closing (should be part of the region move due to balance, also need master side log to confirm), when flushing its memstoreSize is normal(256.2M). And why the flush due to closing can succeed is because it's triggered directly without polling from flushQueue. This can be another side evidence why fdbb3242d3b673bbe4790a47bc30576f is not flushed not because its memstoreSize <=0

        So the root cause (very likely) should be for some reason the flushEntry can't be polled out from flushQueue(or entry added to regionsInQueue but failed to be added to flushQueue, or flushEntry is consumed without triggering flush task but failed to remove according entry in regionsInQueue), and the regionsInQueue still thinks it contains entry for this region(hence reject all later flush requests from 'too many logs' and periodicFlusher). But a manual designated move for this problematic region can succeed since preflush/flush of closing trigger flush directly without polling from flushQueue.

        Show
        Honghua Feng added a comment - Some progress and conclusion till now for the problematic region*fdbb3242d3b673bbe4790a47bc30576f*(lack some log/info, can deduce only by code and log at hand): internalFlushcache() is never called, so root cause won't be its memstoreSize is mis-calculated ('memstoreSize <= 0' check is within internalFlushcache) On contrary memstoreSize should be > 0, and there is flushEntry in regionsInQueue, but never pop up to be executed Why it wasn't moved to other RS is not because it can't be flushed, but because HMaster didn't choose it to move out during balance... (this needs master log to confirm) Below are the deduction process, a bit long... HRegion's private field 'updatesLock': its writeLock().lock() is only called in internalFlushcache(), and after below log. LOG.debug( "Started memstore flush for " + this + ", current region memstore size " +... For the problematic region, above log never be printed, so writeLock().lock() is never called => the call of lock(updatesLock.readLock(), xxx) in write operations won't fail, this means RegionTooBusyException can't be thrown in below method private void lock( final Lock lock, final int multiplier) throws RegionTooBusyException, InterruptedIOException { try { final long waitTime = Math .min(maxBusyWaitDuration, busyWaitDuration * Math .min(multiplier, maxBusyWaitMultiplier)); if (!lock.tryLock(waitTime, TimeUnit.MILLISECONDS)) { throw new RegionTooBusyException( "failed to get a lock in " + waitTime + " ms. " + "regionName=" + ( this .getRegionInfo() == null ? "unknown" : RegionTooBusyException occurs in two places, so it must be thrown here: private void checkResources() throws RegionTooBusyException { // If catalog region, do not impose resource constraints or block updates. if ( this .getRegionInfo().isMetaRegion()) return ; if ( this .memstoreSize.get() > this .blockingMemStoreSize) { requestFlush(); throw new RegionTooBusyException( "Above memstore limit, " + "regionName=" + ( this .getRegionInfo() == null ? "unknown" : this means memstoreSize > 0(also means it shouldn't <= 0 in internalFlushcache()) In above code, requestFlush() is called each time before RegionTooBusyException is thrown, the requestFlush()'s implementation: private void requestFlush() { if ( this .rsServices == null ) { return ; } synchronized (writestate) { if ( this .writestate.isFlushRequested()) { return ; } writestate.flushRequested = true ; } // Make request outside of synchronize block; HBASE-818. this .rsServices.getFlushRequester().requestFlush( this ); if (LOG.isDebugEnabled()) { LOG.debug( "Flush requested on " + this ); } } requestFlush is called the same times as RegionTooBusyException is thrown, but "Flush requested on" log for this region is printed only once...so in this method, it must return before the LOG.debug(), since rsServices is not null, so it must by this.writestate.isFlushRequested() is true => writestate.flushRequested should always be true each time RegionTooBusyException is thrown "Flush requested on" log does be printed once for the problematic region, means 'flushRequested' ever be set to true. And HRegion.writestate.flushRequested is set back to false only in below code besides region initialization public boolean flushcache() throws IOException { ... try { boolean result = internalFlushcache(status); if (coprocessorHost != null ) { status.setStatus( "Running post-flush coprocessor hooks" ); coprocessorHost.postFlush(); } status.markComplete( "Flush successful" ); return result; } finally { synchronized (writestate) { writestate.flushing = false ; this .writestate.flushRequested = false ; writestate.notifyAll(); } } } This means above 'try' block never be called, otherwise the flushRequested in finally block should be set to false, above HRegion.flushcache method is the critical path for flush task coming out of flushQueue(another places calling internalFlushcache are replay split log and when closing region) ==> internalFlushcache() is never called for flush task of this problematic region from flushQueue , this also means the root cause is not 'memstoreSize <=0' (which is checked within internalFlushcache), so adding log before this statement won't help for diagnose even it reproduces next time 'forcing flush of' / 'periodicFlusher requesting flush for' means requestFlush/requestDelayedFlush are called, they all do nothing since "if (!regionsInQueue.containsKey(r))" fail, which means regionsInQueue still contains this region another problematic region is 6b788c498503ddd3e1433a4cd3fb4e39 , its sympotom is exactly the same as fdbb3242d3b673bbe4790a47bc30576f , except it's flushed at '2014-02-11 09:51:11' due to closing (should be part of the region move due to balance, also need master side log to confirm), when flushing its memstoreSize is normal(256.2M). And why the flush due to closing can succeed is because it's triggered directly without polling from flushQueue . This can be another side evidence why fdbb3242d3b673bbe4790a47bc30576f is not flushed not because its memstoreSize <=0 So the root cause (very likely) should be for some reason the flushEntry can't be polled out from flushQueue(or entry added to regionsInQueue but failed to be added to flushQueue, or flushEntry is consumed without triggering flush task but failed to remove according entry in regionsInQueue), and the regionsInQueue still thinks it contains entry for this region(hence reject all later flush requests from 'too many logs' and periodicFlusher). But a manual designated move for this problematic region can succeed since preflush/flush of closing trigger flush directly without polling from flushQueue.
        Hide
        Honghua Feng added a comment -

        But a manual designated move for this problematic region can succeed since preflush/flush of closing trigger flush directly without polling from flushQueue

        A manual flush (from client/shell) is also OK here(without polling from flushQueue) and more lightweight since without closing and re-opening

        Show
        Honghua Feng added a comment - But a manual designated move for this problematic region can succeed since preflush/flush of closing trigger flush directly without polling from flushQueue A manual flush (from client/shell) is also OK here(without polling from flushQueue) and more lightweight since without closing and re-opening
        Hide
        Ted Yu added a comment -

        @Ram:
        Can you enable "hbase.regionserver.servlet.show.queuedump" so that you can get dump of flushQueue ?

        Show
        Ted Yu added a comment - @Ram: Can you enable "hbase.regionserver.servlet.show.queuedump" so that you can get dump of flushQueue ?
        Hide
        ramkrishna.s.vasudevan added a comment -

        Can you enable "hbase.regionserver.servlet.show.queuedump" so that you can get dump of flushQueue ?

        This cluster is stopped now. So next time when i run i can enable this.
        Honghua Feng
        Thanks for the analysis. I was actually analysing another part of the code to see if there is any chance where the flushQueue is not able to pick up the region as you said and mainly due to the equals and compareTo method in FlushRegionEntry. But could not find any proper evidence. What I was thiniking was from time 2014-02-11 08:44:32,433 to 2014-02-11 08:44:34,033 there were lot of flush requests, so may be some problem with the way the region was picked up based on the time based comparsion in the queue. But later felt that because every time there was a flush request.
        The HRegion.shouldFlush() does not check for the size instead only checks for the time of the oldest edit. Also the HLog rollWriter() also checks if there was an oldest edit. So this is what made me think may be the flush size is calculated wrong.
        I will try to get the master logs. The cluster is stopped now but i think the logs should be available.

        Show
        ramkrishna.s.vasudevan added a comment - Can you enable "hbase.regionserver.servlet.show.queuedump" so that you can get dump of flushQueue ? This cluster is stopped now. So next time when i run i can enable this. Honghua Feng Thanks for the analysis. I was actually analysing another part of the code to see if there is any chance where the flushQueue is not able to pick up the region as you said and mainly due to the equals and compareTo method in FlushRegionEntry. But could not find any proper evidence. What I was thiniking was from time 2014-02-11 08:44:32,433 to 2014-02-11 08:44:34,033 there were lot of flush requests, so may be some problem with the way the region was picked up based on the time based comparsion in the queue. But later felt that because every time there was a flush request. The HRegion.shouldFlush() does not check for the size instead only checks for the time of the oldest edit. Also the HLog rollWriter() also checks if there was an oldest edit. So this is what made me think may be the flush size is calculated wrong. I will try to get the master logs. The cluster is stopped now but i think the logs should be available.
        Hide
        Honghua Feng added a comment -

        mainly due to the equals and compareTo method in FlushRegionEntry

        I agree with you on this: equals/compareTo should use things such as region.hashCode() to make entries unique when getDelay of two entries is equal(make equals/compareTo not to return 0/true), since flushQueue(DelayQueue) uses priority queue internally hosting flush entry.

        I will try to get the master logs. The cluster is stopped now but i think the logs should be available.

        Thanks. I just want master logs to confirm another problematic region 6b788c498503ddd3e1433a4cd3fb4e39 was closed due to master selected it to move out when do balance.

        Below are facts for region fdbb3242d3b673bbe4790a47bc30576f I can be sure after digging yesterday:

        1. Its memstoreSize > 0 (RegionTooBusyException is thrown due to this fact)
        2. internalFlushcache never be called
        3. writestate.flushRequested is true (prevent printing 'Flush requested on' log for each write operation before throwing RegionTooBusyException)

        Below are high possible but not confirmed yet:

        1. regionsInQueue contains map entry for this region (reject flush entries generated by rollWriter/periodicFlusher)
        2. no flush entry ever polled out for this region from flushQueue, or polled out once but failed to trigger internalFlushcache and remove corresponding map entry from regionsInQueue (the former is more likely)
        Show
        Honghua Feng added a comment - mainly due to the equals and compareTo method in FlushRegionEntry I agree with you on this: equals/compareTo should use things such as region.hashCode() to make entries unique when getDelay of two entries is equal(make equals/compareTo not to return 0/true), since flushQueue(DelayQueue) uses priority queue internally hosting flush entry. I will try to get the master logs. The cluster is stopped now but i think the logs should be available. Thanks. I just want master logs to confirm another problematic region 6b788c498503ddd3e1433a4cd3fb4e39 was closed due to master selected it to move out when do balance. Below are facts for region fdbb3242d3b673bbe4790a47bc30576f I can be sure after digging yesterday: Its memstoreSize > 0 (RegionTooBusyException is thrown due to this fact) internalFlushcache never be called writestate.flushRequested is true (prevent printing 'Flush requested on' log for each write operation before throwing RegionTooBusyException) Below are high possible but not confirmed yet: regionsInQueue contains map entry for this region (reject flush entries generated by rollWriter/periodicFlusher) no flush entry ever polled out for this region from flushQueue, or polled out once but failed to trigger internalFlushcache and remove corresponding map entry from regionsInQueue (the former is more likely)
        Hide
        Honghua Feng added a comment -

        The HRegion.shouldFlush() does not check for the size instead only checks for the time of the oldest edit. Also the HLog rollWriter() also checks if there was an oldest edit

        If my understanding is correct, these are by design. These two are both to flush regions with old enough edits even their memstoreSize is below flush threshold(to reduce the number of log files at the cost of hfiles with small sizes), so it's natural they don't check whether memstoreSize > someValue(they assume and tolerate memstoreSize with small size). And a region will be selected to flush by rollWriter/periodicalFlusher only when they (do) have some 'old' edits, so the region's memstoreSize must not <= 0(this is an implicit invariant and can be asserted, but not be check-and-exit)

        Show
        Honghua Feng added a comment - The HRegion.shouldFlush() does not check for the size instead only checks for the time of the oldest edit. Also the HLog rollWriter() also checks if there was an oldest edit If my understanding is correct, these are by design. These two are both to flush regions with old enough edits even their memstoreSize is below flush threshold(to reduce the number of log files at the cost of hfiles with small sizes), so it's natural they don't check whether memstoreSize > someValue(they assume and tolerate memstoreSize with small size). And a region will be selected to flush by rollWriter/periodicalFlusher only when they (do) have some 'old' edits, so the region's memstoreSize must not <= 0(this is an implicit invariant and can be asserted, but not be check-and-exit)
        Hide
        ramkrishna.s.vasudevan added a comment -

        Master logs that you asked for Honghua Feng. For the problematic region i could not find anything specific there.

        Show
        ramkrishna.s.vasudevan added a comment - Master logs that you asked for Honghua Feng . For the problematic region i could not find anything specific there.
        Hide
        Honghua Feng added a comment -

        Thanks ramkrishna.s.vasudevan

        There are two problematic regions : fdbb3242d3b673bbe4790a47bc30576f and 6b788c498503ddd3e1433a4cd3fb4e39 — both are requested to flush from time to time but no flush ever happened within [2014-02-11 08:44, 2014-02-11 09:51]

        But at 2014-02-11 09:51:11, 6b788c498503ddd3e1433a4cd3fb4e39 is flushed due to a close operation, and I confirm in the master log you provided just now that the close is part of a move operation triggered by master when it does balance — 6b788c498503ddd3e1433a4cd3fb4e39 is flushed because it is chosen as a move target for balance and flushed during closing.

        While fdbb3242d3b673bbe4790a47bc30576f is not chosen as a move target for balance by master(also can be confirmed in the master log), that's why it doesn't be flushed all along.

        If we suppose these two regions have the same problem regarding non-flush-able internally by regionserver, and 6b788c498503ddd3e1433a4cd3fb4e39 is successfully flushed by a sub-phase of close operation because flush as part of close doesn't bother flushQueue, and the flush log shows that the memstore of 6b788c498503ddd3e1433a4cd3fb4e39 is 256.2M when flushing it. That's a side evidence that fdbb3242d3b673bbe4790a47bc30576f's memstore is also > 0, it's not flushed just because its flush entry can't be polled out of flushQueue...

        Show
        Honghua Feng added a comment - Thanks ramkrishna.s.vasudevan There are two problematic regions : fdbb3242d3b673bbe4790a47bc30576f and 6b788c498503ddd3e1433a4cd3fb4e39 — both are requested to flush from time to time but no flush ever happened within [2014-02-11 08:44, 2014-02-11 09:51] But at 2014-02-11 09:51:11, 6b788c498503ddd3e1433a4cd3fb4e39 is flushed due to a close operation, and I confirm in the master log you provided just now that the close is part of a move operation triggered by master when it does balance — 6b788c498503ddd3e1433a4cd3fb4e39 is flushed because it is chosen as a move target for balance and flushed during closing. While fdbb3242d3b673bbe4790a47bc30576f is not chosen as a move target for balance by master(also can be confirmed in the master log), that's why it doesn't be flushed all along. If we suppose these two regions have the same problem regarding non-flush-able internally by regionserver, and 6b788c498503ddd3e1433a4cd3fb4e39 is successfully flushed by a sub-phase of close operation because flush as part of close doesn't bother flushQueue, and the flush log shows that the memstore of 6b788c498503ddd3e1433a4cd3fb4e39 is 256.2M when flushing it. That's a side evidence that fdbb3242d3b673bbe4790a47bc30576f's memstore is also > 0, it's not flushed just because its flush entry can't be polled out of flushQueue...
        Hide
        Honghua Feng added a comment -

        log for 6b788c498503ddd3e1433a4cd3fb4e39 in regionserver and master, log for fdbb3242d3b673bbe4790a47bc30576f in regionserver and master are attached for reference.

        You can find the similarity(before master balance) and difference(after master balance) between these two regions.

        6b788c498503ddd3e1433a4cd3fb4e39 would be the problematic region if it wasn't chosen as a move target when master did balance...

        Show
        Honghua Feng added a comment - log for 6b788c498503ddd3e1433a4cd3fb4e39 in regionserver and master, log for fdbb3242d3b673bbe4790a47bc30576f in regionserver and master are attached for reference. You can find the similarity(before master balance) and difference(after master balance) between these two regions. 6b788c498503ddd3e1433a4cd3fb4e39 would be the problematic region if it wasn't chosen as a move target when master did balance...
        Hide
        ramkrishna.s.vasudevan added a comment -

        If we suppose these two regions have the same problem regarding non-flush-able internally by regionserver, and 6b788c498503ddd3e1433a4cd3fb4e39 is successfully flushed by a sub-phase of close operation because flush as part of close doesn't bother flushQueue

        This is one thing I have not noticed that while flush happens via close then flushQueue is not used. By design that makes sense.
        One thing I wanted to raise a JIRA is that should flushregionentry use EnvirnonmentEdge.currentMillis instead of System.currentMilllis. I don't know how far it benefits but I think that can be done.
        So there is some reason why flushQueue is not picking up that region?

        Show
        ramkrishna.s.vasudevan added a comment - If we suppose these two regions have the same problem regarding non-flush-able internally by regionserver, and 6b788c498503ddd3e1433a4cd3fb4e39 is successfully flushed by a sub-phase of close operation because flush as part of close doesn't bother flushQueue This is one thing I have not noticed that while flush happens via close then flushQueue is not used. By design that makes sense. One thing I wanted to raise a JIRA is that should flushregionentry use EnvirnonmentEdge.currentMillis instead of System.currentMilllis. I don't know how far it benefits but I think that can be done. So there is some reason why flushQueue is not picking up that region?
        Hide
        Honghua Feng added a comment -

        So there is some reason why flushQueue is not picking up that region?

        I think yes. It's sure RegionTooBusyException is thrown due to 'memstoreSize > blockingMemStoreSize' (see my above deduction process), that means it can't be the case that 'memstoreSize <= 0' (and the previously problematic region 6b788c498503ddd3e1433a4cd3fb4e39 has memstore > 0 (256.2M) when flushing)

        Another two things for sure are:

        1. internalFlushcache was never called;
        2. writestate.flushRequested was never reset to false after it's set to true

        And one thing is highly likely (but can't prove directly by code and log at hand) is regionsInQueue holds entry for region fdbb3242d3b673bbe4790a47bc30576f, since all later flush requests generated by rollWriter and periodicFlusher are rejected by (!regionsInQueue.containsKey(r))

        Show
        Honghua Feng added a comment - So there is some reason why flushQueue is not picking up that region? I think yes . It's sure RegionTooBusyException is thrown due to 'memstoreSize > blockingMemStoreSize' (see my above deduction process), that means it can't be the case that 'memstoreSize <= 0' (and the previously problematic region 6b788c498503ddd3e1433a4cd3fb4e39 has memstore > 0 (256.2M) when flushing) Another two things for sure are: internalFlushcache was never called; writestate.flushRequested was never reset to false after it's set to true And one thing is highly likely (but can't prove directly by code and log at hand ) is regionsInQueue holds entry for region fdbb3242d3b673bbe4790a47bc30576f, since all later flush requests generated by rollWriter and periodicFlusher are rejected by (!regionsInQueue.containsKey(r))
        Hide
        Honghua Feng added a comment -

        One thing I wanted to raise a JIRA is that should flushregionentry use EnvirnonmentEdge.currentMillis instead of System.currentMilllis. I don't know how far it benefits but I think that can be done.

        Wonder if this JIRA is related to this. But if you want to raise a JIRA to replace System.currentMilllis with EnvirnonmentEdge.currentMillis, it's better to do such replace for all occurrences in HBase in a systematic way, since such replace can be done unconditionally without difference, right?

        Show
        Honghua Feng added a comment - One thing I wanted to raise a JIRA is that should flushregionentry use EnvirnonmentEdge.currentMillis instead of System.currentMilllis. I don't know how far it benefits but I think that can be done. Wonder if this JIRA is related to this. But if you want to raise a JIRA to replace System.currentMilllis with EnvirnonmentEdge.currentMillis, it's better to do such replace for all occurrences in HBase in a systematic way, since such replace can be done unconditionally without difference, right?
        Hide
        Himanshu Vashishtha added a comment -

        Yeah, doesn't look like a memstore size being 0 issue.

        How many FlushHandlers were there ?

        Show
        Himanshu Vashishtha added a comment - Yeah, doesn't look like a memstore size being 0 issue. How many FlushHandlers were there ?
        Hide
        Honghua Feng added a comment -

        How many FlushHandlers were there ?

        Should be 1, from the log we can see next flush was started only after the previous one finished, and considering this occurred under heavy write scenario, sounds impossible for multiple concurrent FlushHandlers to run strictly serially by chance all the time. Certainly ramkrishna.s.vasudevan can confirm this

        Show
        Honghua Feng added a comment - How many FlushHandlers were there ? Should be 1, from the log we can see next flush was started only after the previous one finished, and considering this occurred under heavy write scenario, sounds impossible for multiple concurrent FlushHandlers to run strictly serially by chance all the time. Certainly ramkrishna.s.vasudevan can confirm this
        Hide
        ramkrishna.s.vasudevan added a comment -

        Honghua Feng
        The number of flushers should be the default value. I did not change that. Sorry for the late reply.

        But if you want to raise a JIRA to replace System.currentMilllis with EnvirnonmentEdge.currentMillis

        Yes better to change. I am not saying this JIRA is because of this, but just wanted to ensure we change it. Will raise one.

        Show
        ramkrishna.s.vasudevan added a comment - Honghua Feng The number of flushers should be the default value. I did not change that. Sorry for the late reply. But if you want to raise a JIRA to replace System.currentMilllis with EnvirnonmentEdge.currentMillis Yes better to change. I am not saying this JIRA is because of this, but just wanted to ensure we change it. Will raise one.
        Hide
        Andrew Purtell added a comment -

        Where are we with this issue?

        Show
        Andrew Purtell added a comment - Where are we with this issue?
        Hide
        Matt Kapilevich added a comment -

        We're also hitting this issue. The RegionServer is unable to flush one of the regions. I am seeing this in the logs:

        2014-07-24 14:58:42,970 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed compaction: Request = regionName=users,53400000000000000000000000000000000000,1404907131059.6eae263f1f0fd698cca73964118e16b0., storeName=ids, fileCount=3, fileSize=185.0 M, priority=6, time=2961355376528391; duration=7sec
        2014-07-24 14:58:44,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 9557
        2014-07-24 14:58:54,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 9316
        2014-07-24 14:59:04,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 7454
        2014-07-24 14:59:14,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 11197
        2014-07-24 14:59:24,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 19777
        2014-07-24 14:59:34,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 20814
        2014-07-24 14:59:44,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 22542
        2014-07-24 14:59:54,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 16096
        2014-07-24 15:00:04,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 21794
        2014-07-24 15:00:14,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 20873
        

        The unfortunate thing here is that the RegionServer never recovers, even after we stop doing Puts.

        We're on 0.96.1.1-cdh5.0.2

        Are there any workarounds for this? Would setting hbase.hstore.flusher.count to 2 help? (I'm running with default 1).

        Show
        Matt Kapilevich added a comment - We're also hitting this issue. The RegionServer is unable to flush one of the regions. I am seeing this in the logs: 2014-07-24 14:58:42,970 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed compaction: Request = regionName=users,53400000000000000000000000000000000000,1404907131059.6eae263f1f0fd698cca73964118e16b0., storeName=ids, fileCount=3, fileSize=185.0 M, priority=6, time=2961355376528391; duration=7sec 2014-07-24 14:58:44,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 9557 2014-07-24 14:58:54,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 9316 2014-07-24 14:59:04,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 7454 2014-07-24 14:59:14,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 11197 2014-07-24 14:59:24,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 19777 2014-07-24 14:59:34,778 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 20814 2014-07-24 14:59:44,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 22542 2014-07-24 14:59:54,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 16096 2014-07-24 15:00:04,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 21794 2014-07-24 15:00:14,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region users,87080000000000000000000000000000000000,1404907131062.2a8dd55c1011e92f4e32d705196f1e15. after a delay of 20873 The unfortunate thing here is that the RegionServer never recovers, even after we stop doing Puts. We're on 0.96.1.1-cdh5.0.2 Are there any workarounds for this? Would setting hbase.hstore.flusher.count to 2 help? (I'm running with default 1).
        Hide
        Matt Kapilevich added a comment -

        I've attached https://issues.apache.org/jira/secure/attachment/12658147/compaction-queue.png

        It shows that once the RS hits the condition, the compactions just begin to queue up.

        Show
        Matt Kapilevich added a comment - I've attached https://issues.apache.org/jira/secure/attachment/12658147/compaction-queue.png It shows that once the RS hits the condition, the compactions just begin to queue up.
        Hide
        Andrew Purtell added a comment -

        Would setting hbase.hstore.flusher.count to 2 help? (I'm running with default 1).

        That seems a reasonable stab in the dark Matt Kapilevich. It won't address the underlying issue I suspect but may move it out for you.

        Show
        Andrew Purtell added a comment - Would setting hbase.hstore.flusher.count to 2 help? (I'm running with default 1). That seems a reasonable stab in the dark Matt Kapilevich . It won't address the underlying issue I suspect but may move it out for you.
        Hide
        John King added a comment -

        Hi, All

        I also hit the issue yesterday. I think Honghua Feng's analysis is totally right.
        The fqe entry in flushQueue may have been removed before being polled out by the MemStoreFlusher.

        Here are some code from MemStoreFlusher:

        [CODE-BEGIN]
        class MemStoreFlusher implements FlushRequester {

        ...

        private boolean flushRegion(final HRegion region, final boolean emergencyFlush) {
        synchronized (this.regionsInQueue) {
        FlushRegionEntry fqe = this.regionsInQueue.remove(region);
        if (fqe != null && emergencyFlush)

        { // Need to remove from region from delay queue. When NOT an // emergencyFlush, then item was removed via a flushQueue.poll. flushQueue.remove(fqe); }

        }

        ...

        }

        ...

        static class FlushRegionEntry implements FlushQueueEntry {

        ...

        @Override
        public int compareTo(Delayed other)

        { return Long.valueOf(getDelay(TimeUnit.MILLISECONDS) - other.getDelay(TimeUnit.MILLISECONDS)).intValue(); }

        @Override
        public boolean equals(Object obj) {
        if (this == obj)

        { return true; }

        if (obj == null || getClass() != obj.getClass())

        { return false; }

        Delayed other = (Delayed) obj;
        return compareTo(other) == 0;
        }

        ...
        }
        }
        [CODE-END]

        From the code, we can tell:

        1. Two FlushRegionEntry instances were thought be equal only when they have the same delay time.
        2. When emergencyFlush was true, an entry may be removed from the flushQueue.

        Then, if two different fqe (say "A" and "B") have exactly the same delay time, and "B" was flushed as an emergencyFlush,
        "A" may be removed from the fulshQueue instead of "B".

        Because RegionA.writestate.isFlushRequested() always return true and RegionA was still in (MemStoreFlusher).regionsInQueue,
        both (MemStoreFlusher).requestFlush (HRegion r) and (MemStoreFlusher).requestDelayedFlush(HRegion r, long randomDelay) cannot submit RegionA
        into flushQueue. That's why PeriodicMemstoreFlusher and (HRegion).requestFlush () cannot work.

        The problematic region can only be flushed when

        1. Region server's low water mark reached;
        2. Client side RPC (Flush/Split/Merge);
        3. Buckload completting;
        4. Snapshot taking;
        5. Region closing.
        ...

        which may cause the online region refused to serve writting request for a long time.

        PS: Sorry, it's my first time use confluence, my comment may seems a little mess.

        Show
        John King added a comment - Hi, All I also hit the issue yesterday. I think Honghua Feng's analysis is totally right. The fqe entry in flushQueue may have been removed before being polled out by the MemStoreFlusher. Here are some code from MemStoreFlusher: [CODE-BEGIN] class MemStoreFlusher implements FlushRequester { ... private boolean flushRegion(final HRegion region, final boolean emergencyFlush) { synchronized (this.regionsInQueue) { FlushRegionEntry fqe = this.regionsInQueue.remove(region); if (fqe != null && emergencyFlush) { // Need to remove from region from delay queue. When NOT an // emergencyFlush, then item was removed via a flushQueue.poll. flushQueue.remove(fqe); } } ... } ... static class FlushRegionEntry implements FlushQueueEntry { ... @Override public int compareTo(Delayed other) { return Long.valueOf(getDelay(TimeUnit.MILLISECONDS) - other.getDelay(TimeUnit.MILLISECONDS)).intValue(); } @Override public boolean equals(Object obj) { if (this == obj) { return true; } if (obj == null || getClass() != obj.getClass()) { return false; } Delayed other = (Delayed) obj; return compareTo(other) == 0; } ... } } [CODE-END] From the code, we can tell: 1. Two FlushRegionEntry instances were thought be equal only when they have the same delay time. 2. When emergencyFlush was true, an entry may be removed from the flushQueue. Then, if two different fqe (say "A" and "B") have exactly the same delay time, and "B" was flushed as an emergencyFlush, "A" may be removed from the fulshQueue instead of "B". Because RegionA.writestate.isFlushRequested() always return true and RegionA was still in (MemStoreFlusher).regionsInQueue, both (MemStoreFlusher).requestFlush (HRegion r) and (MemStoreFlusher).requestDelayedFlush(HRegion r, long randomDelay) cannot submit RegionA into flushQueue. That's why PeriodicMemstoreFlusher and (HRegion).requestFlush () cannot work. The problematic region can only be flushed when 1. Region server's low water mark reached; 2. Client side RPC (Flush/Split/Merge); 3. Buckload completting; 4. Snapshot taking; 5. Region closing. ... which may cause the online region refused to serve writting request for a long time. PS: Sorry, it's my first time use confluence, my comment may seems a little mess.
        Hide
        John King added a comment -

        And for the problematic region, only when Region server's low water mark reached, can cause region in (MemStoreFlusher).regionsInQueue dequeue.
        i.e. type "flush" in HBase shell cannot resolve this issue.

        Show
        John King added a comment - And for the problematic region, only when Region server's low water mark reached, can cause region in (MemStoreFlusher).regionsInQueue dequeue. i.e. type "flush" in HBase shell cannot resolve this issue.
        Hide
        Ted Yu added a comment -

        Tentative patch which uses region's hashcode as tie-breaker.

        Show
        Ted Yu added a comment - Tentative patch which uses region's hashcode as tie-breaker.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12692895/10499-v2.txt
        against master branch at commit 092c91eb0fc2a6b4044183e9ece71dd03711045d.
        ATTACHMENT ID: 12692895

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestFlushRegionEntry

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692895/10499-v2.txt against master branch at commit 092c91eb0fc2a6b4044183e9ece71dd03711045d. ATTACHMENT ID: 12692895 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestFlushRegionEntry Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12498//console This message is automatically generated.
        Hide
        John King added a comment -

        Hi,Ted
        In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.

        Show
        John King added a comment - Hi,Ted In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.
        Hide
        John King added a comment -

        Hi,Ted
        In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.

        Show
        John King added a comment - Hi,Ted In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.
        Hide
        John King added a comment -

        Hi,Ted
        In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.

        Show
        John King added a comment - Hi,Ted In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.
        Hide
        John King added a comment -

        Hi,Ted
        In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.

        Show
        John King added a comment - Hi,Ted In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.
        Hide
        John King added a comment -

        Hi,Ted
        In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.

        Show
        John King added a comment - Hi,Ted In your v3 patch, you used a strong type conversion, which converted from Delay to FRE, I think it should be FQE.
        Hide
        Ted Yu added a comment -

        @John:
        I get your point

        Show
        Ted Yu added a comment - @John: I get your point
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12692899/10499-v4.txt
        against master branch at commit 092c91eb0fc2a6b4044183e9ece71dd03711045d.
        ATTACHMENT ID: 12692899

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692899/10499-v4.txt against master branch at commit 092c91eb0fc2a6b4044183e9ece71dd03711045d. ATTACHMENT ID: 12692899 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12500//console This message is automatically generated.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Going thro the comments again and the excellent analysis by Honghua Feng and John King I can see that the log that comes out of global heap pressure

        2014-02-11 08:44:30,580 INFO  [Thread-18] regionserver.MemStoreFlusher: Flush of region usertable,user1854,1392107806972.e001ae4de867efc4c4c092e7e5a204e1. due to global heap pressure
        

        Does not appear for the problematic regions. So this would ensure that we have not entered the removal from FQE for these regions.

              if (fqe != null && emergencyFlush) {
                // Need to remove from region from delay queue.  When NOT an
                // emergencyFlush, then item was removed via a flushQueue.poll.
                flushQueue.remove(fqe);
             }
        

        which means they had been removed from FQE but not from regionQueue in such a way that FQE was wrongly mapped to an element which was not removed from the regionQueue. So if region B had to be flushed and it was removed from regionQueue, but from FQE region A got removed.
        So the delay is calculated

            @Override
            public long getDelay(TimeUnit unit) {
              return unit.convert(this.whenToExpire - EnvironmentEdgeManager.currentTime(),
                  TimeUnit.MILLISECONDS);
            }
        

        where whenToExpire is updated in two places, creation of the FRE which is actually synchronized with the regionQueue. So Ideally we would not have two regions with the same whenToExpire (considering there is no precision issue) and in requeue(). The requeue() could update the whenToExpire to the same ts. But here it has not happened.
        If we are going to change the implementation of the equals() in FQE I would suggest compare if the region is the same and then go for the getDelay check in the compareTo()

        Show
        ramkrishna.s.vasudevan added a comment - Going thro the comments again and the excellent analysis by Honghua Feng and John King I can see that the log that comes out of global heap pressure 2014-02-11 08:44:30,580 INFO [ Thread -18] regionserver.MemStoreFlusher: Flush of region usertable,user1854,1392107806972.e001ae4de867efc4c4c092e7e5a204e1. due to global heap pressure Does not appear for the problematic regions. So this would ensure that we have not entered the removal from FQE for these regions. if (fqe != null && emergencyFlush) { // Need to remove from region from delay queue. When NOT an // emergencyFlush, then item was removed via a flushQueue.poll. flushQueue.remove(fqe); } which means they had been removed from FQE but not from regionQueue in such a way that FQE was wrongly mapped to an element which was not removed from the regionQueue. So if region B had to be flushed and it was removed from regionQueue, but from FQE region A got removed. So the delay is calculated @Override public long getDelay(TimeUnit unit) { return unit.convert( this .whenToExpire - EnvironmentEdgeManager.currentTime(), TimeUnit.MILLISECONDS); } where whenToExpire is updated in two places, creation of the FRE which is actually synchronized with the regionQueue. So Ideally we would not have two regions with the same whenToExpire (considering there is no precision issue) and in requeue(). The requeue() could update the whenToExpire to the same ts. But here it has not happened. If we are going to change the implementation of the equals() in FQE I would suggest compare if the region is the same and then go for the getDelay check in the compareTo()
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12693025/HBASE-10499_v5.patch
        against master branch at commit 03e17168c3feab765fec26693318f4b8ae6a9468.
        ATTACHMENT ID: 12693025

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestFlushRegionEntry

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693025/HBASE-10499_v5.patch against master branch at commit 03e17168c3feab765fec26693318f4b8ae6a9468. ATTACHMENT ID: 12693025 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestFlushRegionEntry Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12508//console This message is automatically generated.
        Hide
        ramkrishna.s.vasudevan added a comment -

        For the test case failure- we may need to add the change as in Ted's patch.

        Show
        ramkrishna.s.vasudevan added a comment - For the test case failure- we may need to add the change as in Ted's patch.
        Hide
        Ted Yu added a comment -

        Thanks for the comment Ram.

        FlushQueueEntry#equals() calls FlushQueueEntry#compareTo().
        So I think patch v4 should achieve the goal of making sure the correct region is removed from the queue.

        Show
        Ted Yu added a comment - Thanks for the comment Ram. FlushQueueEntry#equals() calls FlushQueueEntry#compareTo(). So I think patch v4 should achieve the goal of making sure the correct region is removed from the queue.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Ya, but I don't think we need to calculate the hashCode after finding the delay. We should be sure that we are calculating the equals/compareTo for the same region and then find the delay. Also finding hashCode() would involve more operations also. This is my thought.

        Show
        ramkrishna.s.vasudevan added a comment - Ya, but I don't think we need to calculate the hashCode after finding the delay. We should be sure that we are calculating the equals/compareTo for the same region and then find the delay. Also finding hashCode() would involve more operations also. This is my thought.
        Hide
        John King added a comment -

        I think Ram's patch is simpler. And simple is always good.
        The regression failed because one test case in /hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java need to be modified.
        The test case was designed for the old equals implementation.

        Show
        John King added a comment - I think Ram's patch is simpler. And simple is always good. The regression failed because one test case in /hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java need to be modified. The test case was designed for the old equals implementation.
        Hide
        Ted Yu added a comment -

        The contract of the Comparable interface recommends that natural orderings be consistent with equals.

        Show
        Ted Yu added a comment - The contract of the Comparable interface recommends that natural orderings be consistent with equals.
        Show
        Ted Yu added a comment - Please also take a look at http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/concurrent/DelayQueue.java#116
        Hide
        John King added a comment -

        Hi Ted, your point is very reasonable.

        Show
        John King added a comment - Hi Ted, your point is very reasonable.
        Hide
        John King added a comment -

        +1 for patch "10499-v4.txt"

        Show
        John King added a comment - +1 for patch "10499-v4.txt"
        Hide
        Anoop Sam John added a comment -

        int getHashcode();

        Ted, need to add such a new method to interface? Or just call hasCode() from compareTo() and make sure to implement hashCode() in FlushRegionEntry? Just asking.

        Patch looks good.

        Show
        Anoop Sam John added a comment - int getHashcode(); Ted, need to add such a new method to interface? Or just call hasCode() from compareTo() and make sure to implement hashCode() in FlushRegionEntry? Just asking. Patch looks good.
        Hide
        Ted Yu added a comment - - edited

        Thanks for taking a look, Anoop.
        Comparable, Delayed and FlushQueueEntry are all interfaces. So hashcode method has to be added - it is just a matter of the method name.

        Aligning with Java's convention would be nice. Patch v6 does that.

        Show
        Ted Yu added a comment - - edited Thanks for taking a look, Anoop. Comparable, Delayed and FlushQueueEntry are all interfaces. So hashcode method has to be added - it is just a matter of the method name. Aligning with Java's convention would be nice. Patch v6 does that.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12693910/10499-v6.txt
        against master branch at commit 833feefbf977a8208f8f71f5f3bd9b027d54961f.
        ATTACHMENT ID: 12693910

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        -1 core zombie tests. There are 5 zombie test(s): at org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders.testSeekingOnSample(TestDataBlockEncoders.java:205)
        at org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompactionInternals(TestCacheOnWrite.java:458)
        at org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompaction(TestCacheOnWrite.java:410)
        at org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL.testScannerSelection(TestScannerSelectionUsingTTL.java:118)
        at org.apache.hadoop.hbase.TestAcidGuarantees.testScanAtomicity(TestAcidGuarantees.java:354)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693910/10499-v6.txt against master branch at commit 833feefbf977a8208f8f71f5f3bd9b027d54961f. ATTACHMENT ID: 12693910 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . -1 core zombie tests . There are 5 zombie test(s): at org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders.testSeekingOnSample(TestDataBlockEncoders.java:205) at org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompactionInternals(TestCacheOnWrite.java:458) at org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompaction(TestCacheOnWrite.java:410) at org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL.testScannerSelection(TestScannerSelectionUsingTTL.java:118) at org.apache.hadoop.hbase.TestAcidGuarantees.testScanAtomicity(TestAcidGuarantees.java:354) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12548//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        The tests reported hanging actually passed:

        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.363 sec - in org.apache.hadoop.hbase.TestAcidGuarantees
        

        See also the summary:

        +1 core tests. The patch passed unit tests in .
        

        Local run of TestDataBlockEncoders, TestCacheOnWrite, TestScannerSelectionUsingTTL and TestAcidGuarantees passed too.

        Let me attach patch v6 again.

        Show
        Ted Yu added a comment - The tests reported hanging actually passed: Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.363 sec - in org.apache.hadoop.hbase.TestAcidGuarantees See also the summary: +1 core tests. The patch passed unit tests in . Local run of TestDataBlockEncoders, TestCacheOnWrite, TestScannerSelectionUsingTTL and TestAcidGuarantees passed too. Let me attach patch v6 again.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12693941/10499-v6.txt
        against master branch at commit 319f9bb7918af8cfe7e65f97b654f37f0d5983f3.
        ATTACHMENT ID: 12693941

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:

        -1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testRegionTransitionOperations(TestMasterObserver.java:1604)
        at org.apache.hadoop.hbase.regionserver.TestJoinedScanners.testJoinedScanners(TestJoinedScanners.java:92)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693941/10499-v6.txt against master branch at commit 319f9bb7918af8cfe7e65f97b654f37f0d5983f3. ATTACHMENT ID: 12693941 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 2 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testRegionTransitionOperations(TestMasterObserver.java:1604) at org.apache.hadoop.hbase.regionserver.TestJoinedScanners.testJoinedScanners(TestJoinedScanners.java:92) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12551//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        From QA run #12551 :

         /x1/jenkins/jenkins-home/jobs/PreCommit-HBASE-Build/builds/2015-01-22_18-10-22/archive/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat-output.txt may be corrupted
        	at jsync.protocol.FileSequenceReader.read(FileSequenceReader.java:45)
        	at com.cloudbees.jenkins.plugins.jsync.archiver.JSyncArtifactManager.remoteSync(JSyncArtifactManager.java:145)
        	at com.cloudbees.jenkins.plugins.jsync.archiver.JSyncArtifactManager.archive(JSyncArtifactManager.java:68)
        	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:140)
        	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:756)
        	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
        	at hudson.model.Build$BuildExecution.post2(Build.java:182)
        	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:669)
        	at hudson.model.Run.execute(Run.java:1731)
        	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        	at hudson.model.ResourceController.execute(ResourceController.java:88)
        	at hudson.model.Executor.run(Executor.java:232)
        Recording test results
        [description-setter] Could not determine description.
        Finished: FAILURE
        

        The above indicated problem with build infrastructure.

        Show
        Ted Yu added a comment - From QA run #12551 : /x1/jenkins/jenkins-home/jobs/PreCommit-HBASE-Build/builds/2015-01-22_18-10-22/archive/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat-output.txt may be corrupted at jsync.protocol.FileSequenceReader.read(FileSequenceReader.java:45) at com.cloudbees.jenkins.plugins.jsync.archiver.JSyncArtifactManager.remoteSync(JSyncArtifactManager.java:145) at com.cloudbees.jenkins.plugins.jsync.archiver.JSyncArtifactManager.archive(JSyncArtifactManager.java:68) at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:140) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:756) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720) at hudson.model.Build$BuildExecution.post2(Build.java:182) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:669) at hudson.model.Run.execute(Run.java:1731) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:232) Recording test results [description-setter] Could not determine description. Finished: FAILURE The above indicated problem with build infrastructure.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12693983/10499-v7.txt
        against master branch at commit 319f9bb7918af8cfe7e65f97b654f37f0d5983f3.
        ATTACHMENT ID: 12693983

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693983/10499-v7.txt against master branch at commit 319f9bb7918af8cfe7e65f97b654f37f0d5983f3. ATTACHMENT ID: 12693983 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12559//console This message is automatically generated.
        Hide
        Anoop Sam John added a comment -

        +1

        Show
        Anoop Sam John added a comment - +1
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12694093/10499-v8.txt
        against master branch at commit 5fbf80ee5ecb288804d2d2d042199dcd834ae848.
        ATTACHMENT ID: 12694093

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12694093/10499-v8.txt against master branch at commit 5fbf80ee5ecb288804d2d2d042199dcd834ae848. ATTACHMENT ID: 12694093 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12567//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Thanks for the reviews - they're really helpful.

        Will commit later today.

        Show
        Ted Yu added a comment - Thanks for the reviews - they're really helpful. Will commit later today.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #6049 (See https://builds.apache.org/job/HBase-TRUNK/6049/)
        HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 74adb11f4c504abbb6a52de72b53883ad7b952b4)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6049 (See https://builds.apache.org/job/HBase-TRUNK/6049/ ) HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 74adb11f4c504abbb6a52de72b53883ad7b952b4) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-1.1 #102 (See https://builds.apache.org/job/HBase-1.1/102/)
        HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 3a529c04cebb4f3debdfd42fb00d3736dc2ea2fd)

        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-1.1 #102 (See https://builds.apache.org/job/HBase-1.1/102/ ) HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 3a529c04cebb4f3debdfd42fb00d3736dc2ea2fd) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        Hide
        Andrew Purtell added a comment -

        This was reported against 0.98, shouldn't this be fixed in/for 0.98?

        Show
        Andrew Purtell added a comment - This was reported against 0.98, shouldn't this be fixed in/for 0.98?
        Hide
        Andrew Purtell added a comment -

        Let me see about 0.98.

        Show
        Andrew Purtell added a comment - Let me see about 0.98.
        Hide
        Ted Yu added a comment -

        Patch for 0.98 adds the TestFlushRegionEntry which was absent in 0.98

        Show
        Ted Yu added a comment - Patch for 0.98 adds the TestFlushRegionEntry which was absent in 0.98
        Hide
        Andrew Purtell added a comment - - edited

        Thanks Ted Yu, I applied the 0.98 patch and checked the new test and other flush tests, looks good, +1

        Show
        Andrew Purtell added a comment - - edited Thanks Ted Yu , I applied the 0.98 patch and checked the new test and other flush tests, looks good, +1
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-1.0 #679 (See https://builds.apache.org/job/HBase-1.0/679/)
        HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 973961b23fc6f3e4748ae7a213c9a89ff89dbb33)

        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-1.0 #679 (See https://builds.apache.org/job/HBase-1.0/679/ ) HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 973961b23fc6f3e4748ae7a213c9a89ff89dbb33) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        Hide
        ramkrishna vasudevan added a comment -

        +1 on patch
        https://issues.apache.org/jira/browse/HBASE-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
        ]
        RegionTooBusyException
        ----------------------------------------------------------------------------------------------
        10499-v6.txt, HBASE-10499.patch, HBASE-10499_v5.patch,
        compaction-queue.png, hbase-root-master-ip-10-157-0-229.zip,
        hbase-root-regionserver-ip-10-93-128-92.zip, master_4e39.log,
        master_576f.log, rs_4e39.log, rs_576f.log, t1.dump, t2.dump,
        workloada_0.98.dat
        this version. Doesn't seem so to me.
        200 regions. In one of the run with 0.98 server and 0.98 client I faced
        this problem like the hlogs became more and the system requested flushes
        for those many regions.
        remained unflushed. The ripple effect of this on the client side
        org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
        54 actions: RegionTooBusyException: 54 times,
        org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
        54 actions: RegionTooBusyException: 54 times,
        org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:187)
        org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:171)
        org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:897)
        org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:961)
        org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1225)
        Too many hlogs: logs=38, maxlogs=32; forcing flush of 23 regions(s):
        97d8ae2f78910cc5ded5fbb1ddad8492, d396b8a1da05c871edcb68a15608fdf2,
        01a68742a1be3a9705d574ad68fec1d7, 1250381046301e7465b6cf398759378e,
        127c133f47d0419bd5ab66675aff76d4, 9f01c5d25ddc6675f750968873721253,
        29c055b5690839c2fa357cd8e871741e, ca4e33e3eb0d5f8314ff9a870fc43463,
        acfc6ae756e193b58d956cb71ccf0aa3, 187ea304069bc2a3c825bc10a59c7e84,
        0ea411edc32d5c924d04bf126fa52d1e, e2f9331fc7208b1b230a24045f3c869e,
        d9309ca864055eddf766a330352efc7a, 1a71bdf457288d449050141b5ff00c69,
        0ba9089db28e977f86a27f90bbab9717, fdbb3242d3b673bbe4790a47bc30576f,
        bbadaa1f0e62d8a8650080b824187850, b1a5de30d8603bd5d9022e09c574501b,
        cc6a9fabe44347ed65e7c325faa72030, 313b17dbff2497f5041b57fe13fa651e,
        6b788c498503ddd3e1433a4cd3fb4e39, 3d71274fe4f815882e9626e1cfa050d1,
        acc43e4b42c1a041078774f4f20a3ff5
        Too many hlogs: logs=53, maxlogs=32; forcing flush of 2 regions(s):
        fdbb3242d3b673bbe4790a47bc30576f, 6b788c498503ddd3e1433a4cd3fb4e39
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
        delay of 16689
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
        delay of 15868
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
        delay of 20847
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
        delay of 20099
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
        delay of 8677
        Too many hlogs: logs=54, maxlogs=32; forcing flush of 1 regions(s):
        fdbb3242d3b673bbe4790a47bc30576f
        regions but this region stays with the RS that has this issue. One
        important observation is that in HRegion.internalflushCache() we need to
        add a debug log here
        not happen and no logs related to flush are printed in the logs. so due to
        some reason this memstore.size() has become 0( I assume this). The earlier
        bugs were also due to similar reason.

        Show
        ramkrishna vasudevan added a comment - +1 on patch https://issues.apache.org/jira/browse/HBASE-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RegionTooBusyException ---------------------------------------------------------------------------------------------- 10499-v6.txt, HBASE-10499 .patch, HBASE-10499 _v5.patch, compaction-queue.png, hbase-root-master-ip-10-157-0-229.zip, hbase-root-regionserver-ip-10-93-128-92.zip, master_4e39.log, master_576f.log, rs_4e39.log, rs_576f.log, t1.dump, t2.dump, workloada_0.98.dat this version. Doesn't seem so to me. 200 regions. In one of the run with 0.98 server and 0.98 client I faced this problem like the hlogs became more and the system requested flushes for those many regions. remained unflushed. The ripple effect of this on the client side org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 54 actions: RegionTooBusyException: 54 times, org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 54 actions: RegionTooBusyException: 54 times, org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:187) org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:171) org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:897) org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:961) org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1225) Too many hlogs: logs=38, maxlogs=32; forcing flush of 23 regions(s): 97d8ae2f78910cc5ded5fbb1ddad8492, d396b8a1da05c871edcb68a15608fdf2, 01a68742a1be3a9705d574ad68fec1d7, 1250381046301e7465b6cf398759378e, 127c133f47d0419bd5ab66675aff76d4, 9f01c5d25ddc6675f750968873721253, 29c055b5690839c2fa357cd8e871741e, ca4e33e3eb0d5f8314ff9a870fc43463, acfc6ae756e193b58d956cb71ccf0aa3, 187ea304069bc2a3c825bc10a59c7e84, 0ea411edc32d5c924d04bf126fa52d1e, e2f9331fc7208b1b230a24045f3c869e, d9309ca864055eddf766a330352efc7a, 1a71bdf457288d449050141b5ff00c69, 0ba9089db28e977f86a27f90bbab9717, fdbb3242d3b673bbe4790a47bc30576f, bbadaa1f0e62d8a8650080b824187850, b1a5de30d8603bd5d9022e09c574501b, cc6a9fabe44347ed65e7c325faa72030, 313b17dbff2497f5041b57fe13fa651e, 6b788c498503ddd3e1433a4cd3fb4e39, 3d71274fe4f815882e9626e1cfa050d1, acc43e4b42c1a041078774f4f20a3ff5 Too many hlogs: logs=53, maxlogs=32; forcing flush of 2 regions(s): fdbb3242d3b673bbe4790a47bc30576f, 6b788c498503ddd3e1433a4cd3fb4e39 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 16689 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a delay of 15868 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 20847 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a delay of 20099 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 8677 Too many hlogs: logs=54, maxlogs=32; forcing flush of 1 regions(s): fdbb3242d3b673bbe4790a47bc30576f regions but this region stays with the RS that has this issue. One important observation is that in HRegion.internalflushCache() we need to add a debug log here not happen and no logs related to flush are printed in the logs. so due to some reason this memstore.size() has become 0( I assume this). The earlier bugs were also due to similar reason.
        Hide
        ramkrishna vasudevan added a comment -

        https://issues.apache.org/jira/browse/HBASE-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
        ]
        causing RegionTooBusyException
        ----------------------------------------------------------------------------------------------
        10499-v6.txt, HBASE-10499.patch, HBASE-10499_v5.patch,
        compaction-queue.png, hbase-root-master-ip-10-157-0-229.zip,
        hbase-root-regionserver-ip-10-93-128-92.zip, master_4e39.log,
        master_576f.log, rs_4e39.log, rs_576f.log, t1.dump, t2.dump,
        workloada_0.98.dat
        to this version. Doesn't seem so to me.
        has 200 regions. In one of the run with 0.98 server and 0.98 client I
        faced this problem like the hlogs became more and the system requested
        flushes for those many regions.
        remained unflushed. The ripple effect of this on the client side
        org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
        54 actions: RegionTooBusyException: 54 times,
        org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
        54 actions: RegionTooBusyException: 54 times,
        org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:187)
        org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:171)
        org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:897)
        org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:961)
        org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1225)
        wal.FSHLog: Too many hlogs: logs=38, maxlogs=32; forcing flush of 23
        regions(s): 97d8ae2f78910cc5ded5fbb1ddad8492,
        d396b8a1da05c871edcb68a15608fdf2, 01a68742a1be3a9705d574ad68fec1d7,
        1250381046301e7465b6cf398759378e, 127c133f47d0419bd5ab66675aff76d4,
        9f01c5d25ddc6675f750968873721253, 29c055b5690839c2fa357cd8e871741e,
        ca4e33e3eb0d5f8314ff9a870fc43463, acfc6ae756e193b58d956cb71ccf0aa3,
        187ea304069bc2a3c825bc10a59c7e84, 0ea411edc32d5c924d04bf126fa52d1e,
        e2f9331fc7208b1b230a24045f3c869e, d9309ca864055eddf766a330352efc7a,
        1a71bdf457288d449050141b5ff00c69, 0ba9089db28e977f86a27f90bbab9717,
        fdbb3242d3b673bbe4790a47bc30576f, bbadaa1f0e62d8a8650080b824187850,
        b1a5de30d8603bd5d9022e09c574501b, cc6a9fabe44347ed65e7c325faa72030,
        313b17dbff2497f5041b57fe13fa651e, 6b788c498503ddd3e1433a4cd3fb4e39,
        3d71274fe4f815882e9626e1cfa050d1, acc43e4b42c1a041078774f4f20a3ff5
        wal.FSHLog: Too many hlogs: logs=53, maxlogs=32; forcing flush of 2
        regions(s): fdbb3242d3b673bbe4790a47bc30576f,
        6b788c498503ddd3e1433a4cd3fb4e39
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
        delay of 16689
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
        delay of 15868
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
        delay of 20847
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
        delay of 20099
        regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
        flush for region
        usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
        delay of 8677
        wal.FSHLog: Too many hlogs: logs=54, maxlogs=32; forcing flush of 1
        regions(s): fdbb3242d3b673bbe4790a47bc30576f
        regions but this region stays with the RS that has this issue. One
        important observation is that in HRegion.internalflushCache() we need to
        add a debug log here
        does not happen and no logs related to flush are printed in the logs. so
        due to some reason this memstore.size() has become 0( I assume this). The
        earlier bugs were also due to similar reason.

        Show
        ramkrishna vasudevan added a comment - https://issues.apache.org/jira/browse/HBASE-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] causing RegionTooBusyException ---------------------------------------------------------------------------------------------- 10499-v6.txt, HBASE-10499 .patch, HBASE-10499 _v5.patch, compaction-queue.png, hbase-root-master-ip-10-157-0-229.zip, hbase-root-regionserver-ip-10-93-128-92.zip, master_4e39.log, master_576f.log, rs_4e39.log, rs_576f.log, t1.dump, t2.dump, workloada_0.98.dat to this version. Doesn't seem so to me. has 200 regions. In one of the run with 0.98 server and 0.98 client I faced this problem like the hlogs became more and the system requested flushes for those many regions. remained unflushed. The ripple effect of this on the client side org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 54 actions: RegionTooBusyException: 54 times, org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 54 actions: RegionTooBusyException: 54 times, org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:187) org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:171) org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:897) org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:961) org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1225) wal.FSHLog: Too many hlogs: logs=38, maxlogs=32; forcing flush of 23 regions(s): 97d8ae2f78910cc5ded5fbb1ddad8492, d396b8a1da05c871edcb68a15608fdf2, 01a68742a1be3a9705d574ad68fec1d7, 1250381046301e7465b6cf398759378e, 127c133f47d0419bd5ab66675aff76d4, 9f01c5d25ddc6675f750968873721253, 29c055b5690839c2fa357cd8e871741e, ca4e33e3eb0d5f8314ff9a870fc43463, acfc6ae756e193b58d956cb71ccf0aa3, 187ea304069bc2a3c825bc10a59c7e84, 0ea411edc32d5c924d04bf126fa52d1e, e2f9331fc7208b1b230a24045f3c869e, d9309ca864055eddf766a330352efc7a, 1a71bdf457288d449050141b5ff00c69, 0ba9089db28e977f86a27f90bbab9717, fdbb3242d3b673bbe4790a47bc30576f, bbadaa1f0e62d8a8650080b824187850, b1a5de30d8603bd5d9022e09c574501b, cc6a9fabe44347ed65e7c325faa72030, 313b17dbff2497f5041b57fe13fa651e, 6b788c498503ddd3e1433a4cd3fb4e39, 3d71274fe4f815882e9626e1cfa050d1, acc43e4b42c1a041078774f4f20a3ff5 wal.FSHLog: Too many hlogs: logs=53, maxlogs=32; forcing flush of 2 regions(s): fdbb3242d3b673bbe4790a47bc30576f, 6b788c498503ddd3e1433a4cd3fb4e39 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 16689 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a delay of 15868 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 20847 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a delay of 20099 regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a delay of 8677 wal.FSHLog: Too many hlogs: logs=54, maxlogs=32; forcing flush of 1 regions(s): fdbb3242d3b673bbe4790a47bc30576f regions but this region stays with the RS that has this issue. One important observation is that in HRegion.internalflushCache() we need to add a debug log here does not happen and no logs related to flush are printed in the logs. so due to some reason this memstore.size() has become 0( I assume this). The earlier bugs were also due to similar reason.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #815 (See https://builds.apache.org/job/HBase-0.98/815/)
        HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 5484f7958f8ce929c619c377f07917f05cab0db6)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #815 (See https://builds.apache.org/job/HBase-0.98/815/ ) HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 5484f7958f8ce929c619c377f07917f05cab0db6) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java
        Hide
        ramkrishna.s.vasudevan added a comment -

        Ted, was not able to +1 it on time. I tried +1 from my mobile while on travel from the mail that I received for this JIRA and found that the same got updated only today morning. Thanks for your work and time Ted.

        Show
        ramkrishna.s.vasudevan added a comment - Ted, was not able to +1 it on time. I tried +1 from my mobile while on travel from the mail that I received for this JIRA and found that the same got updated only today morning. Thanks for your work and time Ted.
        Hide
        Ted Yu added a comment -

        Enjoy your trip, Ram.

        Show
        Ted Yu added a comment - Enjoy your trip, Ram.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #776 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/776/)
        HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 5484f7958f8ce929c619c377f07917f05cab0db6)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #776 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/776/ ) HBASE-10499 In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException (Ram and Ted) (tedyu: rev 5484f7958f8ce929c619c377f07917f05cab0db6) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java
        Hide
        Enis Soztutar added a comment -

        Closing this issue after 1.0.0 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 1.0.0 release.

          People

          • Assignee:
            Ted Yu
            Reporter:
            ramkrishna.s.vasudevan
          • Votes:
            0 Vote for this issue
            Watchers:
            21 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development