HBase
  1. HBase
  2. HBASE-7008

Set scanner caching to a better default, disable Nagles

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: Client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This patch increases the default scanner caching to 100.
      It also enables tcpNoDelay by default (i.e. disabled Nagle's algorithm).

      This will generally lead to better performance, but in some situation may lead to different network patterns and increase memory consumption on both the client and the server.
      Show
      This patch increases the default scanner caching to 100. It also enables tcpNoDelay by default (i.e. disabled Nagle's algorithm). This will generally lead to better performance, but in some situation may lead to different network patterns and increase memory consumption on both the client and the server.
    • Tags:
      0.96notable
    1. HBASE-7008-v2.patch
      2 kB
      Liang Xie
    2. HBASE-7008.patch
      1 kB
      Liang Xie
    3. 7008-v4.txt
      5 kB
      Lars Hofhansl
    4. 7008-v3.txt
      3 kB
      Lars Hofhansl
    5. 7008-trunk-v6.txt
      8 kB
      Lars Hofhansl
    6. 7008-trunk-v5.txt
      8 kB
      Lars Hofhansl
    7. 7008-0.94-v3.txt
      7 kB
      Lars Hofhansl
    8. 7008-0.94-v2.txt
      5 kB
      Lars Hofhansl
    9. 7008-0.94.txt
      3 kB
      Lars Hofhansl

      Issue Links

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12549645/HBASE-7008.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 82 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549645/HBASE-7008.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 82 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3075//console This message is automatically generated.
        Hide
        Jieshan Bean added a comment -

        Thanks for the patch, xieliang.
        I suggest to introduce a new member variable in HConstants to define this dafault value. What do you think?

        Show
        Jieshan Bean added a comment - Thanks for the patch, xieliang. I suggest to introduce a new member variable in HConstants to define this dafault value. What do you think?
        Hide
        Liang Xie added a comment -

        emm, IMHO, it's not very necessary

        Show
        Liang Xie added a comment - emm, IMHO, it's not very necessary
        Hide
        Jieshan Bean added a comment -

        Ok, fine. Magic number is always not good, but anyway it's not a problem.

        Show
        Jieshan Bean added a comment - Ok, fine. Magic number is always not good, but anyway it's not a problem.
        Hide
        Lars Hofhansl added a comment -

        Actually this will have no effect since it is also defined in hbase-defaults.xml. It needs to be changed there as well.

        Show
        Lars Hofhansl added a comment - Actually this will have no effect since it is also defined in hbase-defaults.xml. It needs to be changed there as well.
        Hide
        Lars Hofhansl added a comment -

        Since this is a behavioral change 0.94 might not be entirely appropriate. On the other hand, 1 is almost never appropriate.

        Show
        Lars Hofhansl added a comment - Since this is a behavioral change 0.94 might not be entirely appropriate. On the other hand, 1 is almost never appropriate.
        Hide
        stack added a comment -

        I'd be more conservative and do 10 instead of 100.

        Show
        stack added a comment - I'd be more conservative and do 10 instead of 100.
        Hide
        Liang Xie added a comment -

        Addressed Lars & Michael's comments, thanks for having a nice review

        Show
        Liang Xie added a comment - Addressed Lars & Michael's comments, thanks for having a nice review
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12549811/HBASE-7008-v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 82 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
        org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
        org.apache.hadoop.hbase.client.TestFromClientSide

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549811/HBASE-7008-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 82 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor org.apache.hadoop.hbase.client.TestFromClientSide Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3087//console This message is automatically generated.
        Hide
        Lars Hofhansl added a comment -

        OK. Let's leave it at 10. (I still think 100 is better default, though.)

        While we're at it, wanna make a constant (in HConstants) for hbase.client.scanner.caching and for the 10? I can do that on commit too.

        Show
        Lars Hofhansl added a comment - OK. Let's leave it at 10. (I still think 100 is better default, though.) While we're at it, wanna make a constant (in HConstants) for hbase.client.scanner.caching and for the 10? I can do that on commit too.
        Hide
        Lars Hofhansl added a comment -

        Hmm... Just noticed we set the scanner caching for meta scans to 100 by default. In that case, though, we know that the values scanned will be small.

        Show
        Lars Hofhansl added a comment - Hmm... Just noticed we set the scanner caching for meta scans to 100 by default. In that case, though, we know that the values scanned will be small.
        Hide
        Lars Hofhansl added a comment -

        Same patch with constants.

        Show
        Lars Hofhansl added a comment - Same patch with constants.
        Hide
        Lars Hofhansl added a comment -

        0.94 patch.

        Going to commit later today, unless there are objections.

        Show
        Lars Hofhansl added a comment - 0.94 patch. Going to commit later today, unless there are objections.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12550187/7008-0.94.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3107//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550187/7008-0.94.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3107//console This message is automatically generated.
        Hide
        Lars Hofhansl added a comment -

        This actually causes TestFromClientSide#testScanMetrics to fail, since the metrics are only collected when roundtrips occur.

        Show
        Lars Hofhansl added a comment - This actually causes TestFromClientSide#testScanMetrics to fail, since the metrics are only collected when roundtrips occur.
        Hide
        Lars Hofhansl added a comment -

        Trunk patch with test fix, also add a few more lines to the test to validate metrics are in fact not collected unless we cause a roundtrip.

        Show
        Lars Hofhansl added a comment - Trunk patch with test fix, also add a few more lines to the test to validate metrics are in fact not collected unless we cause a roundtrip.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12550215/7008-v4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 82 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550215/7008-v4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 82 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3109//console This message is automatically generated.
        Hide
        Lars Hofhansl added a comment -

        And a 0.94 version with test fixed.

        Show
        Lars Hofhansl added a comment - And a 0.94 version with test fixed.
        Hide
        Jean-Daniel Cryans added a comment -

        Could a bigger default with a hbase.client.scanner.max.result.size that doesn't default to Long.MAX_VALUE be more appropriate?

        Show
        Jean-Daniel Cryans added a comment - Could a bigger default with a hbase.client.scanner.max.result.size that doesn't default to Long.MAX_VALUE be more appropriate?
        Hide
        stack added a comment -

        Chatting w/ LarsH, we should be doing size-based rather than count of items. This'll do for now. +1 on patch (even if it super conservative – Thanks Lars).

        Show
        stack added a comment - Chatting w/ LarsH, we should be doing size-based rather than count of items. This'll do for now. +1 on patch (even if it super conservative – Thanks Lars).
        Hide
        Andrew Purtell added a comment -

        Sorry I'm late to the party. 10 is better than 1, but 100 seems better than that IMHO. Just $0.02.

        Show
        Andrew Purtell added a comment - Sorry I'm late to the party. 10 is better than 1, but 100 seems better than that IMHO. Just $0.02.
        Hide
        Lars Hofhansl added a comment -

        I'd prefer 100 too. Let's make it 100 in trunk and 10 in 0.94... Just to be safe?

        Show
        Lars Hofhansl added a comment - I'd prefer 100 too. Let's make it 100 in trunk and 10 in 0.94... Just to be safe?
        Hide
        stack added a comment -

        I can bend on the 100 since three folks say it. Just trying to be conservative (it doesn't suit me well I can tell). Sorry for making extra work around a small change.

        Show
        stack added a comment - I can bend on the 100 since three folks say it. Just trying to be conservative (it doesn't suit me well I can tell). Sorry for making extra work around a small change.
        Hide
        Lars Hofhansl added a comment - - edited

        So while I was testing performance of 10 vs 100 with various KV sizes I ran into HBASE-5943. I would like to fold that in with this patch. There is no reason to use Nagles with HBase.

        Edit: The usual spelling/grammar mistakes

        Show
        Lars Hofhansl added a comment - - edited So while I was testing performance of 10 vs 100 with various KV sizes I ran into HBASE-5943 . I would like to fold that in with this patch. There is no reason to use Nagles with HBase. Edit: The usual spelling/grammar mistakes
        Hide
        Lars Hofhansl added a comment - - edited

        Did some quick unscientific testing (with Nagles disabled!):

        rows with 1 column (100 bytes):

        Caching ms/row
        1 0.17
        10 0.03
        100 0.006
        1000 0.003
        2000 0.003
        5000 0.002
        10000 0.002

        rows with 10 columns (100 bytes each) in a single column family:

        Caching ms/row
        1 0.3
        10 0.2 **
        100 0.05
        1000 0.03
        2000 0.02
        5000 0.02
        10000 0.02

        (**) Must have crossed a packet size boundary (localhost packet size is 65k)

        rows with 10 columns (1000 bytes each) in a single column family:

        Caching ms/row
        1 1
        10 0.5
        100 0.2
        1000 0.09
        2000 0.08
        5000 0.07
        10000 0.1

        rows with 100 columns (1000 bytes each) in a single column family:

        Caching ms/row
        1 3
        10 1.6
        100 0.8
        1000 0.7
        2000 **

        (**) Crashed the RegionServer in the default setting (OOM?)

        rows with 100 columns (10000 bytes each - so 1mb rows) in a single column family:

        Caching ms/row
        1 18
        10 8
        100 8
        1000 **

        (**) Anything over 150 crashed the RegionServer in the default setting (OOM?)

        This was all done with client and server on the same machine, so worst case for a larger cache setting. If the client is further away these effects will be much more pronounced.

        So it seems a 100 would be a good default value to start with.

        Edit: Remeasured some of the numbers.

        Show
        Lars Hofhansl added a comment - - edited Did some quick unscientific testing (with Nagles disabled!): rows with 1 column (100 bytes): Caching ms/row 1 0.17 10 0.03 100 0.006 1000 0.003 2000 0.003 5000 0.002 10000 0.002 rows with 10 columns (100 bytes each) in a single column family: Caching ms/row 1 0.3 10 0.2 ** 100 0.05 1000 0.03 2000 0.02 5000 0.02 10000 0.02 (**) Must have crossed a packet size boundary (localhost packet size is 65k) rows with 10 columns (1000 bytes each) in a single column family: Caching ms/row 1 1 10 0.5 100 0.2 1000 0.09 2000 0.08 5000 0.07 10000 0.1 rows with 100 columns (1000 bytes each) in a single column family: Caching ms/row 1 3 10 1.6 100 0.8 1000 0.7 2000 ** (**) Crashed the RegionServer in the default setting (OOM?) rows with 100 columns (10000 bytes each - so 1mb rows) in a single column family: Caching ms/row 1 18 10 8 100 8 1000 ** (**) Anything over 150 crashed the RegionServer in the default setting (OOM?) This was all done with client and server on the same machine, so worst case for a larger cache setting. If the client is further away these effects will be much more pronounced. So it seems a 100 would be a good default value to start with. Edit: Remeasured some of the numbers.
        Hide
        Lars Hofhansl added a comment -

        Should say I tested this with a local HDFS setup (HBase fully distributed, but on a single machine, the block cache was disabled)

        Show
        Lars Hofhansl added a comment - Should say I tested this with a local HDFS setup (HBase fully distributed, but on a single machine, the block cache was disabled)
        Hide
        Lars Hofhansl added a comment -

        Sames with Nagles left on:
        tows with 1 column (100 bytes):

        Caching ms/row
        1 0.17
        10 0.003
        100 0.04 **
        1000 0.004
        2000 0.003
        5000 0.003
        10000 0.003

        (**) Strange Nagles interaction here

        rows with 10 columns (100 bytes each) in a single column family:

        Caching ms/row
        1 0.8
        10 4 **
        100 0.3
        1000 0.002
        2000 0.002
        5000 0.002
        10000 0.002

        (**) Strange Nagles interaction here

        rows with 10 columns (1000 bytes each) in a single column family:

        Caching ms/row
        1 40 **
        10 1.7
        100 0.3
        1000 0.08
        2000 0.08
        5000 0.07
        10000 0.1

        (**) Nagles

        rows with 100 columns (1000 bytes each) in a single column family:

        Caching ms/row
        1 8
        10 3
        100 0.9
        1000 0.7
        2000 **

        (**) Crashed the RegionServer in the default setting (OOM?)

        rows with 100 columns (10000 bytes each - so 1mb rows) in a single column family:

        Caching ms/row
        1 60
        10 12
        100 8
        1000 **

        (**) Anything over 150 crashed the RegionServer in the default setting (OOM?)

        So disabling Nagels is not always better!

        Show
        Lars Hofhansl added a comment - Sames with Nagles left on: tows with 1 column (100 bytes): Caching ms/row 1 0.17 10 0.003 100 0.04 ** 1000 0.004 2000 0.003 5000 0.003 10000 0.003 (**) Strange Nagles interaction here rows with 10 columns (100 bytes each) in a single column family: Caching ms/row 1 0.8 10 4 ** 100 0.3 1000 0.002 2000 0.002 5000 0.002 10000 0.002 (**) Strange Nagles interaction here rows with 10 columns (1000 bytes each) in a single column family: Caching ms/row 1 40 ** 10 1.7 100 0.3 1000 0.08 2000 0.08 5000 0.07 10000 0.1 (**) Nagles rows with 100 columns (1000 bytes each) in a single column family: Caching ms/row 1 8 10 3 100 0.9 1000 0.7 2000 ** (**) Crashed the RegionServer in the default setting (OOM?) rows with 100 columns (10000 bytes each - so 1mb rows) in a single column family: Caching ms/row 1 60 10 12 100 8 1000 ** (**) Anything over 150 crashed the RegionServer in the default setting (OOM?) So disabling Nagels is not always better!
        Hide
        stack added a comment -

        Does having larger value get us over nagle effect and strangeness?

        Show
        stack added a comment - Does having larger value get us over nagle effect and strangeness?
        Hide
        Lars Hofhansl added a comment -

        It does look that way. It also seems that disabling Nagle's almost never leads to worse scan performance.
        So we have various options now:

        1. Disable Nagle's and increase scanner caching to 100 by default
        2. Only increase the scanner caching default, leave Nagle's on by default
        3. Leave things as they are and document setting scanner caching in the important settings.

        Based on my limited testing I'd vote for #1.

        Show
        Lars Hofhansl added a comment - It does look that way. It also seems that disabling Nagle's almost never leads to worse scan performance. So we have various options now: Disable Nagle's and increase scanner caching to 100 by default Only increase the scanner caching default, leave Nagle's on by default Leave things as they are and document setting scanner caching in the important settings. Based on my limited testing I'd vote for #1.
        Hide
        Lars Hofhansl added a comment -

        I could also be convinced that #3 is a good option, as long as we actually list the exact config option that need to be changed. I'd be happy to write a doc entry for that.

        Show
        Lars Hofhansl added a comment - I could also be convinced that #3 is a good option, as long as we actually list the exact config option that need to be changed. I'd be happy to write a doc entry for that.
        Hide
        stack added a comment -

        I could go for #1. I just added to the doc. a link your fancy graphs above into the nagle's section.... so we have #3 now I'd argue (look in src – haven't pushed the site yet)

        Show
        stack added a comment - I could go for #1. I just added to the doc. a link your fancy graphs above into the nagle's section.... so we have #3 now I'd argue (look in src – haven't pushed the site yet)
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3488 (See https://builds.apache.org/job/HBase-TRUNK/3488/)
        Add link to Lars's graphs in hbase-7008 where he plays w/ nagles and different data sizes (Revision 1402399)

        Result = FAILURE

        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3488 (See https://builds.apache.org/job/HBase-TRUNK/3488/ ) Add link to Lars's graphs in hbase-7008 where he plays w/ nagles and different data sizes (Revision 1402399) Result = FAILURE
        Hide
        Nicolas Liochon added a comment -

        I think #1 is better than #3, because with #1 it will be used by everybody, hence it will becomes production-proven for most use cases. It's as well better for people new to HBase.

        Just my 2 cents

        Show
        Nicolas Liochon added a comment - I think #1 is better than #3, because with #1 it will be used by everybody, hence it will becomes production-proven for most use cases. It's as well better for people new to HBase. Just my 2 cents
        Hide
        Liang Xie added a comment -

        +1 for #1, the default setting should be user-friendly for most of end-users or newbie
        we should not expect end-users tuning per doc always. Or in other words, if we leave all those things unchanged, maybe there'll be still many users complain why HBase performance so bad, or only the experts could finger the root cause out.

        Show
        Liang Xie added a comment - +1 for #1, the default setting should be user-friendly for most of end-users or newbie we should not expect end-users tuning per doc always. Or in other words, if we leave all those things unchanged, maybe there'll be still many users complain why HBase performance so bad, or only the experts could finger the root cause out.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #239 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/239/)
        Add link to Lars's graphs in hbase-7008 where he plays w/ nagles and different data sizes (Revision 1402399)

        Result = FAILURE

        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #239 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/239/ ) Add link to Lars's graphs in hbase-7008 where he plays w/ nagles and different data sizes (Revision 1402399) Result = FAILURE
        Hide
        stack added a comment -

        Lets do #1 then. What you think Lars Hofhansl

        Show
        stack added a comment - Lets do #1 then. What you think Lars Hofhansl
        Hide
        Lars Hofhansl added a comment -

        +1 on #1

        Show
        Lars Hofhansl added a comment - +1 on #1
        Hide
        Lars Hofhansl added a comment -

        0.94 patch I plan to commit. Enables tcpnodelay and sets scanner caching default to 100.

        Show
        Lars Hofhansl added a comment - 0.94 patch I plan to commit. Enables tcpnodelay and sets scanner caching default to 100.
        Hide
        Lars Hofhansl added a comment -

        Trunk patch

        Show
        Lars Hofhansl added a comment - Trunk patch
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12551050/7008-trunk-v5.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 85 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestScannerTimeout

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551050/7008-trunk-v5.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 85 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestScannerTimeout Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3162//console This message is automatically generated.
        Hide
        Lars Hofhansl added a comment -

        Trunk version that fixes TestScannerTimeout

        Show
        Lars Hofhansl added a comment - Trunk version that fixes TestScannerTimeout
        Hide
        stack added a comment -

        +1 on patch. This is an incompatible change and release note should note what we've changed. Good stuff Lars.

        Show
        stack added a comment - +1 on patch. This is an incompatible change and release note should note what we've changed. Good stuff Lars.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12551052/7008-trunk-v6.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 85 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551052/7008-trunk-v6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 85 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3163//console This message is automatically generated.
        Hide
        Lars Hofhansl added a comment -

        How's this for a release note?

        Show
        Lars Hofhansl added a comment - How's this for a release note?
        Hide
        stack added a comment -

        +1 on release note.

        Show
        stack added a comment - +1 on release note.
        Hide
        Lars Hofhansl added a comment -

        The release notes made me think... Do we want to change this in a 0.94 point release?
        Definitely for 0.96... Just not sure about 0.94.

        What do people think? Would you find this unexpected if you upgraded from 0.94.2 to 0.94.3?

        Show
        Lars Hofhansl added a comment - The release notes made me think... Do we want to change this in a 0.94 point release? Definitely for 0.96... Just not sure about 0.94. What do people think? Would you find this unexpected if you upgraded from 0.94.2 to 0.94.3?
        Hide
        Ted Yu added a comment -

        Good point, Lars.
        A discussion on dev@hbase ?

        Show
        Ted Yu added a comment - Good point, Lars. A discussion on dev@hbase ?
        Hide
        stack added a comment -

        Should not change in point release I'd say. Shouldn't be messing w/ defaults in a point release.

        Show
        stack added a comment - Should not change in point release I'd say. Shouldn't be messing w/ defaults in a point release.
        Hide
        stack added a comment -

        Lars Hofhansl This is good to commit still? Has +1s and nice release note but not committed. Want me to do it?

        Show
        stack added a comment - Lars Hofhansl This is good to commit still? Has +1s and nice release note but not committed. Want me to do it?
        Hide
        Lars Hofhansl added a comment -

        Oops. Thought I had committed this. Will do later today.

        Show
        Lars Hofhansl added a comment - Oops. Thought I had committed this. Will do later today.
        Hide
        stack added a comment -

        Lars Hofhansl When this goes in, should we close HBASE-5943?

        Show
        stack added a comment - Lars Hofhansl When this goes in, should we close HBASE-5943 ?
        Hide
        Lars Hofhansl added a comment -

        Committed to trunk

        Show
        Lars Hofhansl added a comment - Committed to trunk
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3594 (See https://builds.apache.org/job/HBase-TRUNK/3594/)
        HBASE-7008 Set scanner caching to a better default, disable Nagles (Revision 1417233)

        Result = FAILURE
        larsh :
        Files :

        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
        • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3594 (See https://builds.apache.org/job/HBase-TRUNK/3594/ ) HBASE-7008 Set scanner caching to a better default, disable Nagles (Revision 1417233) Result = FAILURE larsh : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #285 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/285/)
        HBASE-7008 Set scanner caching to a better default, disable Nagles (Revision 1417233)

        Result = FAILURE
        larsh :
        Files :

        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
        • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #285 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/285/ ) HBASE-7008 Set scanner caching to a better default, disable Nagles (Revision 1417233) Result = FAILURE larsh : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java
        Hide
        stack added a comment -

        Marking closed.

        Show
        stack added a comment - Marking closed.

          People

          • Assignee:
            Liang Xie
            Reporter:
            Liang Xie
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development