HBase
  1. HBase
  2. HBASE-4956

Control direct memory buffer consumption by HBaseClient

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Tags:
      0.96notable

      Description

      As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory.

      We should consider using netty for NIO-related tasks.

      1. thread_get.rb
        3 kB
        Bob Copeland
      2. 4956.txt
        0.9 kB
        Ted Yu

        Issue Links

          Activity

          Hide
          Lars Hofhansl added a comment -

          HBASE-4805 should help with too, as it would allow the overall set of threads that any HTable can use to perform IO (at least of batch operations).

          Show
          Lars Hofhansl added a comment - HBASE-4805 should help with too, as it would allow the overall set of threads that any HTable can use to perform IO (at least of batch operations).
          Hide
          Lars Hofhansl added a comment - - edited

          We might not have to go all the way to use netty (although that would be nice).
          If we find that it is possible to avoid calling HBaseClient.Connection.sendParam from the client thread, but have the call actually be made from the Connection thread (after the callable representing the operation was queued), we have limited the number of threads that will have cached DirectBuffer on their behalf.
          Stack suggested the only reason for the direct call might be to pass errors back to the client. We could hand the client a Deferred or Future that will eventually hold any encountered exception, the client could (and would by default to keep the current synchronous behavior) also wait on that object.

          Show
          Lars Hofhansl added a comment - - edited We might not have to go all the way to use netty (although that would be nice). If we find that it is possible to avoid calling HBaseClient.Connection.sendParam from the client thread, but have the call actually be made from the Connection thread (after the callable representing the operation was queued), we have limited the number of threads that will have cached DirectBuffer on their behalf. Stack suggested the only reason for the direct call might be to pass errors back to the client. We could hand the client a Deferred or Future that will eventually hold any encountered exception, the client could (and would by default to keep the current synchronous behavior) also wait on that object.
          Hide
          Ted Yu added a comment -

          Since the proposal involves asynchronous communication, we should devise new API which can be used to validate the reduction in use of direct memory buffer.

          Show
          Ted Yu added a comment - Since the proposal involves asynchronous communication, we should devise new API which can be used to validate the reduction in use of direct memory buffer.
          Hide
          Lars Hofhansl added a comment -

          Just checked the HTable code in trunk... Puts (even single put operations) already go through a thread from the HTable's thread pool. A single get request is issuing a request directly, but get(List<Get>) is also using the pool.
          So limiting the number of threads in a single thread pool and using the new HTable constructor from HBASE-4805, this can be controlled. Need to make one is using only get(List<Get>) in addition to a relatively small global threadpool.
          At Salesforce we'll shoot for pool with ~100 threads and a waiting queue of about 100 as well (for very large clusters that may be too small though).

          Show
          Lars Hofhansl added a comment - Just checked the HTable code in trunk... Puts (even single put operations) already go through a thread from the HTable's thread pool. A single get request is issuing a request directly, but get(List<Get>) is also using the pool. So limiting the number of threads in a single thread pool and using the new HTable constructor from HBASE-4805 , this can be controlled. Need to make one is using only get(List<Get>) in addition to a relatively small global threadpool. At Salesforce we'll shoot for pool with ~100 threads and a waiting queue of about 100 as well (for very large clusters that may be too small though).
          Hide
          stack added a comment -

          @Lars So supposition is that we bound the number of direct buffers client allocates by bounding the number of threads in the thread pool (and have all accesses go via the HTable thread pool)? If so, sounds good. What happens if as many HTable instances as there are application-level threads and say the application spawns lots of threads? The application should then go via HTablePool? Or rather, you are suggesting that application uses new HBASE-4805 api and pass common executor to all HTable instances?

          Show
          stack added a comment - @Lars So supposition is that we bound the number of direct buffers client allocates by bounding the number of threads in the thread pool (and have all accesses go via the HTable thread pool)? If so, sounds good. What happens if as many HTable instances as there are application-level threads and say the application spawns lots of threads? The application should then go via HTablePool? Or rather, you are suggesting that application uses new HBASE-4805 api and pass common executor to all HTable instances?
          Hide
          Ted Yu added a comment -

          Currently lily-project uses a hack to share ExecutorService among all HTable instances.
          Once HBASE-5084 is integrated, client application would have full control over reusing ExecutorService across all HTable instances.

          Show
          Ted Yu added a comment - Currently lily-project uses a hack to share ExecutorService among all HTable instances. Once HBASE-5084 is integrated, client application would have full control over reusing ExecutorService across all HTable instances.
          Hide
          Lars Hofhansl added a comment -

          @Stack and Ted: Yes, if the application uses the API introduced in HBASE-4805 the number of threads doing network ops is limited to a single thread pools (that can be sized accordingly).

          Still not ideal. The regionserver side of things does chunking (8k chunks by default), so even if a lot of buffers are cached for a lot of threads it won't add to much memory - even for 1000 threads each caching 3 buffer you'd only use 24mb, and since they are the same size they are reusable). Maybe the client do the same.

          Show
          Lars Hofhansl added a comment - @Stack and Ted: Yes, if the application uses the API introduced in HBASE-4805 the number of threads doing network ops is limited to a single thread pools (that can be sized accordingly). Still not ideal. The regionserver side of things does chunking (8k chunks by default), so even if a lot of buffers are cached for a lot of threads it won't add to much memory - even for 1000 threads each caching 3 buffer you'd only use 24mb, and since they are the same size they are reusable). Maybe the client do the same.
          Hide
          Lars Hofhansl added a comment -

          Looking at the stacktrace that Jonathan posted to the asynchhabse group:

                  java.nio.Buffer.<init>(Buffer.java:172)
                  java.nio.ByteBuffer.<init>(ByteBuffer.java:259)
                  java.nio.ByteBuffer.<init>(ByteBuffer.java:267)
                  java.nio.MappedByteBuffer.<init>(MappedByteBuffer.java:64)
                  java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
                  java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
                  sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
                  sun.nio.ch.IOUtil.write(IOUtil.java:37)
                  sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
                  org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
                  org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
                  org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
                  org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
          -->     java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
          -->     java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                  java.io.DataOutputStream.flush(DataOutputStream.java:106)
                  org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:518)
                  org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:751)
                  org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
                  $Proxy11.getProtocolVersion(<Unknown Source>:Unknown line) 
          

          Since the writes go through a BufferOutputStream they are already limited to 8k chunks (default buffer size).
          So I do not really know what the problem is... In order to make this any kind of problem you need have buffers cached for 1m threads (to get to 8GB).
          Anyway with the optional global ExecutorService of HBASE-4805 this should no longer be a problem.

          Show
          Lars Hofhansl added a comment - Looking at the stacktrace that Jonathan posted to the asynchhabse group: java.nio.Buffer.<init>(Buffer.java:172) java.nio.ByteBuffer.<init>(ByteBuffer.java:259) java.nio.ByteBuffer.<init>(ByteBuffer.java:267) java.nio.MappedByteBuffer.<init>(MappedByteBuffer.java:64) java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97) java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155) sun.nio.ch.IOUtil.write(IOUtil.java:37) sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) --> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) --> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) java.io.DataOutputStream.flush(DataOutputStream.java:106) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:518) org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:751) org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) $Proxy11.getProtocolVersion(<Unknown Source>:Unknown line) Since the writes go through a BufferOutputStream they are already limited to 8k chunks (default buffer size). So I do not really know what the problem is... In order to make this any kind of problem you need have buffers cached for 1m threads (to get to 8GB). Anyway with the optional global ExecutorService of HBASE-4805 this should no longer be a problem.
          Hide
          stack added a comment -

          Hmmm... Looking in code, we allocate a BufferedOutputStream using default buffer size:

                  this.out = new DataOutputStream
                      (new BufferedOutputStream(NetUtils.getOutputStream(socket)));
          

          ... so, it seems that yeah, at any one time, you'd think the maximum allocation 8k as you say Lars.

          Jonathan says 3 of these buffers allocated when reading. Presume similar writing (easy to check I suppose. I havent'). Thats 6*8k per thread since these are thread local.... which still don't seem like that much. You'd need lots of threads to run into trouble.

          We need to reproduce.

          Show
          stack added a comment - Hmmm... Looking in code, we allocate a BufferedOutputStream using default buffer size: this .out = new DataOutputStream ( new BufferedOutputStream(NetUtils.getOutputStream(socket))); ... so, it seems that yeah, at any one time, you'd think the maximum allocation 8k as you say Lars. Jonathan says 3 of these buffers allocated when reading. Presume similar writing (easy to check I suppose. I havent'). Thats 6*8k per thread since these are thread local.... which still don't seem like that much. You'd need lots of threads to run into trouble. We need to reproduce.
          Hide
          Bob Copeland added a comment -

          I have this bug on our servers (0.90.4-cdh3). I think Johnathan's analysis was spot on, but it matters much more for read path since readFully() allocates a direct buffer as large as the target read.

          Here's an easy way to reproduce:

          1) create a ~10 meg row in some table
          2) edit bin/hbase script to have max direct memory size < 10m, e.g.:

          +JAVA_HEAP_MAX="-Xmx1000m -XX:MaxDirectMemorySize=1m"
          

          3) now have any client try to read this row. If read side were chunking with < 1m pieces, it should still be able to read.

          Obviously this example is contrived, but we hit the problem with around 1-2000 threads (we have large rows in our DB). Here's what our stack trace looks like:

          java.lang.OutOfMemoryError: Direct buffer memory
                  at java.nio.Bits.reserveMemory(Bits.java:633)
                  at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:98)
                  at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
                  at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
                  at sun.nio.ch.IOUtil.read(IOUtil.java:174)
                  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
                  at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
                  at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
                  at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
                  at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
                  at java.io.FilterInputStream.read(FilterInputStream.java:116)
                  at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:299)
                  at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
                  at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
                  at java.io.DataInputStream.readFully(DataInputStream.java:178)
                  at org.apache.hadoop.hbase.client.Result.readFields(Result.java:504)
                  at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
                  at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:259)
                  at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:554)
                  at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)
          

          The following works around the problem by only doing 8k allocations:

          diff --git a/src/main/java/org/apache/hadoop/hbase/client/Result.java b/src/main/java/org/apache/hadoop/hbase/client/Result.java
          index 8a0c1a9..378ca88 100644
          --- a/src/main/java/org/apache/hadoop/hbase/client/Result.java
          +++ b/src/main/java/org/apache/hadoop/hbase/client/Result.java
          @@ -502,10 +502,18 @@ public class Result implements Writable, WritableWithSize {
                 return;
               }
               byte [] raw = new byte[totalBuffer];
          -    in.readFully(raw, 0, totalBuffer);
          +    readChunked(in, raw, 0, totalBuffer);
               bytes = new ImmutableBytesWritable(raw, 0, totalBuffer);
             }
           
          +  private void readChunked(final DataInput in, byte[] dest, int ofs, int len)
          +  throws IOException {
          +    int maxread = 8192;
          +
          +    for (; ofs < len; ofs += maxread)
          +      in.readFully(dest, ofs, Math.min(len - ofs, maxread));
          +  }
          +
             //Create KeyValue[] when needed
             private void readFields() {
               if (bytes == null) {
          
          Show
          Bob Copeland added a comment - I have this bug on our servers (0.90.4-cdh3). I think Johnathan's analysis was spot on, but it matters much more for read path since readFully() allocates a direct buffer as large as the target read. Here's an easy way to reproduce: 1) create a ~10 meg row in some table 2) edit bin/hbase script to have max direct memory size < 10m, e.g.: +JAVA_HEAP_MAX= "-Xmx1000m -XX:MaxDirectMemorySize=1m" 3) now have any client try to read this row. If read side were chunking with < 1m pieces, it should still be able to read. Obviously this example is contrived, but we hit the problem with around 1-2000 threads (we have large rows in our DB). Here's what our stack trace looks like: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:633) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:98) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155) at sun.nio.ch.IOUtil.read(IOUtil.java:174) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.FilterInputStream.read(FilterInputStream.java:116) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:299) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at java.io.DataInputStream.readFully(DataInputStream.java:178) at org.apache.hadoop.hbase.client.Result.readFields(Result.java:504) at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521) at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:259) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:554) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477) The following works around the problem by only doing 8k allocations: diff --git a/src/main/java/org/apache/hadoop/hbase/client/Result.java b/src/main/java/org/apache/hadoop/hbase/client/Result.java index 8a0c1a9..378ca88 100644 --- a/src/main/java/org/apache/hadoop/hbase/client/Result.java +++ b/src/main/java/org/apache/hadoop/hbase/client/Result.java @@ -502,10 +502,18 @@ public class Result implements Writable, WritableWithSize { return ; } byte [] raw = new byte [totalBuffer]; - in.readFully(raw, 0, totalBuffer); + readChunked(in, raw, 0, totalBuffer); bytes = new ImmutableBytesWritable(raw, 0, totalBuffer); } + private void readChunked( final DataInput in, byte [] dest, int ofs, int len) + throws IOException { + int maxread = 8192; + + for (; ofs < len; ofs += maxread) + in.readFully(dest, ofs, Math .min(len - ofs, maxread)); + } + //Create KeyValue[] when needed private void readFields() { if (bytes == null ) {
          Hide
          Lars Hofhansl added a comment -

          This looks like a good approach. I'd be OK committing this (after passing tests, etc).

          Show
          Lars Hofhansl added a comment - This looks like a good approach. I'd be OK committing this (after passing tests, etc).
          Hide
          Lars Hofhansl added a comment -

          Potentially every call to readFully over the wire is suspect. Just did a quick check. It's used in Put/Append/KeyValue/Bytes/etc/etc.

          Show
          Lars Hofhansl added a comment - Potentially every call to readFully over the wire is suspect. Just did a quick check. It's used in Put/Append/KeyValue/Bytes/etc/etc.
          Hide
          Todd Lipcon added a comment -

          I'd think we should make the max read configurable. Otherwise we should do some benchmarks to show that this doesn't cause a regression. My worry is that this would translate to a bunch more syscalls, no?

          Show
          Todd Lipcon added a comment - I'd think we should make the max read configurable. Otherwise we should do some benchmarks to show that this doesn't cause a regression. My worry is that this would translate to a bunch more syscalls, no?
          Hide
          Lars Hofhansl added a comment -

          Yeah, we should quantify this. 8k seems like a very reasonable chunk size, though (probably anything between 1k and 64k would be OK ).

          @Bob: Any chance you can whip up a quick performance test for this?

          Show
          Lars Hofhansl added a comment - Yeah, we should quantify this. 8k seems like a very reasonable chunk size, though (probably anything between 1k and 64k would be OK ). @Bob: Any chance you can whip up a quick performance test for this?
          Hide
          Bob Copeland added a comment -

          With regard to readFully() being everywhere – perhaps it would be less intrusive to subclass DataInput, or to modify one of the read functions further down in the stack like org.apache.hadoop.net.SocketInputStream.read() to do the chunking. You guys know the code better than I.

          These numbers are for standalone mode. I can't easily do a test on our distributed cluster but I don't think it would change the numbers much. I'll upload my test script and then anyone who wants can repeat the experiment on their setup.

          Each trial consists of a run of "hbase org.jruby.Main thread_get.rb 1", which does 100 gets of a ~10 MB row, 3 times in a row, computing avg & stddev of wall-clock time. Maxread variable is changed for every run.

          size            avg          stddev
          ~~~~         ~~~~~~          ~~~~~~
          1            33.690             .10
          1k            3.072             .18
          2k            3.395             .19
          4k            3.493             .16
          8k            3.306             .19
          16k           2.918             .21
          32k           2.835             .06
          64k           2.689             .08
          128k          3.048             .03
          len           3.328             .08
          
          Show
          Bob Copeland added a comment - With regard to readFully() being everywhere – perhaps it would be less intrusive to subclass DataInput, or to modify one of the read functions further down in the stack like org.apache.hadoop.net.SocketInputStream.read() to do the chunking. You guys know the code better than I. These numbers are for standalone mode. I can't easily do a test on our distributed cluster but I don't think it would change the numbers much. I'll upload my test script and then anyone who wants can repeat the experiment on their setup. Each trial consists of a run of "hbase org.jruby.Main thread_get.rb 1", which does 100 gets of a ~10 MB row, 3 times in a row, computing avg & stddev of wall-clock time. Maxread variable is changed for every run. size avg stddev ~~~~ ~~~~~~ ~~~~~~ 1 33.690 .10 1k 3.072 .18 2k 3.395 .19 4k 3.493 .16 8k 3.306 .19 16k 2.918 .21 32k 2.835 .06 64k 2.689 .08 128k 3.048 .03 len 3.328 .08
          Hide
          Bob Copeland added a comment -

          Test script for reproducing the error (with max direct memory limits in place) or benchmarking different chunk sizes. Although it can do threading, I have mostly passed "1" to do the tests in a single thread.

          Show
          Bob Copeland added a comment - Test script for reproducing the error (with max direct memory limits in place) or benchmarking different chunk sizes. Although it can do threading, I have mostly passed "1" to do the tests in a single thread.
          Hide
          Bob Copeland added a comment -

          Just a follow-up – we've been running with this patch (8k window size) for a couple of months now, and have no more OOMs, and no noticeable performance or functionality regressions.

          Show
          Bob Copeland added a comment - Just a follow-up – we've been running with this patch (8k window size) for a couple of months now, and have no more OOMs, and no noticeable performance or functionality regressions.
          Hide
          Ted Yu added a comment -

          Patch from Bob.

          Show
          Ted Yu added a comment - Patch from Bob.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537187/4956.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537187/4956.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2410//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          Thanks Bob! I think we should commit this. To 0.94 and 0.96 at least.
          If there're no objections I'll do that.

          From your analysis it looks like 32k would be the sweet spot, but since you ran it in production with an 8k window, let's commit this.

          Show
          Lars Hofhansl added a comment - Thanks Bob! I think we should commit this. To 0.94 and 0.96 at least. If there're no objections I'll do that. From your analysis it looks like 32k would be the sweet spot, but since you ran it in production with an 8k window, let's commit this.
          Hide
          Bob Copeland added a comment -

          Yeah, buffer size doesn't really matter to our workload so out of laziness I stuck with 8k, but a larger one should be OK. Worth noting that 8k is better than the full length (i.e. unpatched) in my tests, presumably because of the additional allocation overhead.

          Show
          Bob Copeland added a comment - Yeah, buffer size doesn't really matter to our workload so out of laziness I stuck with 8k, but a larger one should be OK. Worth noting that 8k is better than the full length (i.e. unpatched) in my tests, presumably because of the additional allocation overhead.
          Hide
          Ted Yu added a comment -

          Patch integrated to trunk.

          Thanks for the patch, Bob.

          Thanks for the review, Lars.

          Show
          Ted Yu added a comment - Patch integrated to trunk. Thanks for the patch, Bob. Thanks for the review, Lars.
          Hide
          Ted Yu added a comment -

          Integrated to 0.94 since this JIRA is marked against 0.94.1 release.

          Show
          Ted Yu added a comment - Integrated to 0.94 since this JIRA is marked against 0.94.1 release.
          Hide
          Lars Hofhansl added a comment -

          Thanks Ted.

          Show
          Lars Hofhansl added a comment - Thanks Ted.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3153 (See https://builds.apache.org/job/HBase-TRUNK/3153/)
          HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363526)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3153 (See https://builds.apache.org/job/HBase-TRUNK/3153/ ) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363526) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Hide
          Lars Hofhansl added a comment -

          Added you as contributor Bob.

          Show
          Lars Hofhansl added a comment - Added you as contributor Bob.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #341 (See https://builds.apache.org/job/HBase-0.94/341/)
          HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #341 (See https://builds.apache.org/job/HBase-0.94/341/ ) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #100 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/100/)
          HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363526)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #100 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/100/ ) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363526) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #44 (See https://builds.apache.org/job/HBase-0.94-security/44/)
          HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #44 (See https://builds.apache.org/job/HBase-0.94-security/44/ ) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/)
          HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/ ) HBASE-4956 Control direct memory buffer consumption by HBaseClient (Bob Copeland) (Revision 1363533) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java

            People

            • Assignee:
              Bob Copeland
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development