HBase
  1. HBase
  2. HBASE-11390

PerformanceEvaluation: add an option to use a single connection

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.99.0
    • Fix Version/s: 0.99.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The PE tool uses one connection per client. It does not match some use cases when we have multiple threads sharing the same connection.

      1. 11390.v2.patch
        11 kB
        Nicolas Liochon
      2. 11390.v1.patch
        3 kB
        Nicolas Liochon

        Activity

        Hide
        Matteo Bertozzi added a comment -

        +1

        Show
        Matteo Bertozzi added a comment - +1
        Hide
        Nick Dimiduk added a comment -

        Would be better to getConnection is supposed to be deprecated / go away (HBASE-9117) in favor of explicitly managed connections. Better to call createConnection once and tack it onto the options object.

        Show
        Nick Dimiduk added a comment - Would be better to getConnection is supposed to be deprecated / go away ( HBASE-9117 ) in favor of explicitly managed connections. Better to call createConnection once and tack it onto the options object.
        Hide
        Nicolas Liochon added a comment -

        You mean we should put a Connection obect in the option class?

        Show
        Nicolas Liochon added a comment - You mean we should put a Connection obect in the option class?
        Hide
        Nick Dimiduk added a comment -

        Yes, I think so. It would need to be added post-serialization in MR mode.

        Show
        Nick Dimiduk added a comment - Yes, I think so. It would need to be added post-serialization in MR mode.
        Hide
        Nicolas Liochon added a comment -

        If we go this route, having a static variable in Test would not be better? Then the option class would remain clean?

        Show
        Nicolas Liochon added a comment - If we go this route, having a static variable in Test would not be better? Then the option class would remain clean?
        Hide
        Nick Dimiduk added a comment -

        How about passing the Connection as a constructor parameter to Test instead of the current Configuration?

        Show
        Nick Dimiduk added a comment - How about passing the Connection as a constructor parameter to Test instead of the current Configuration?
        Hide
        Nicolas Liochon added a comment -

        Let me try

        Show
        Nicolas Liochon added a comment - Let me try
        Hide
        Nicolas Liochon added a comment -

        What do you think of the v2?
        As a side note, how can I use the pool in ConnectionManager?

        Show
        Nicolas Liochon added a comment - What do you think of the v2? As a side note, how can I use the pool in ConnectionManager?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12651688/11390.v1.patch
        against trunk revision .
        ATTACHMENT ID: 12651688

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651688/11390.v1.patch against trunk revision . ATTACHMENT ID: 12651688 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9807//console This message is automatically generated.
        Hide
        Nick Dimiduk added a comment -

        Yes, I like this patch better. Thank you. One last question: is there a place to explicitly close the shared connection? I don't see such a place at this time.

        What do you mean re: the pool. You can provide your own and pass it down to the createConnection method.

        Show
        Nick Dimiduk added a comment - Yes, I like this patch better. Thank you. One last question: is there a place to explicitly close the shared connection? I don't see such a place at this time. What do you mean re: the pool. You can provide your own and pass it down to the createConnection method.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12651707/11390.v2.patch
        against trunk revision .
        ATTACHMENT ID: 12651707

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.master.TestSplitLogManager

        -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning(TestHRegion.java:3299)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651707/11390.v2.patch against trunk revision . ATTACHMENT ID: 12651707 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning(TestHRegion.java:3299) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9810//console This message is automatically generated.
        Hide
        Nicolas Liochon added a comment -

        is there a place to explicitly close the shared connection? I don't see such a place at this time.

        There is a close after the call to #test()?

        For the pool, there is a pool in the HCOnnectionImplementation, but it's not usable outside of the package. But I will spend some time on this on a separate issue.

        Show
        Nicolas Liochon added a comment - is there a place to explicitly close the shared connection? I don't see such a place at this time. There is a close after the call to #test()? For the pool, there is a pool in the HCOnnectionImplementation, but it's not usable outside of the package. But I will spend some time on this on a separate issue.
        Hide
        Nicolas Liochon added a comment -

        The tests failure and the findbugs are unrelated.

        Show
        Nicolas Liochon added a comment - The tests failure and the findbugs are unrelated.
        Hide
        Nick Dimiduk added a comment -

        Yes, I see both calls now. +1.

        Show
        Nick Dimiduk added a comment - Yes, I see both calls now. +1.
        Hide
        Nicolas Liochon added a comment -

        Committed to master, thanks for the review, Matteo & Nick!

        Show
        Nicolas Liochon added a comment - Committed to master, thanks for the review, Matteo & Nick!
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #5229 (See https://builds.apache.org/job/HBase-TRUNK/5229/)
        HBASE-11390 PerformanceEvaluation: add an option to use a single connection (nkeywal: rev e991fd8055112e2ddc0bbeade2ef74504e2407dd)

        • hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5229 (See https://builds.apache.org/job/HBase-TRUNK/5229/ ) HBASE-11390 PerformanceEvaluation: add an option to use a single connection (nkeywal: rev e991fd8055112e2ddc0bbeade2ef74504e2407dd) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
        Hide
        Enis Soztutar added a comment -

        Closing this issue after 0.99.0 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

          People

          • Assignee:
            Nicolas Liochon
            Reporter:
            Nicolas Liochon
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development