Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.

      To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
      The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.

      Currently the HTableMultiplexer only supports the put operation.

      1. ASF.LICENSE.NOT.GRANTED--D2775.1.patch
        21 kB
        Phabricator
      2. ASF.LICENSE.NOT.GRANTED--D2775.1.patch
        21 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--D2775.2.patch
        27 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--D2775.2.patch
        27 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--D2775.3.patch
        30 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--D2775.4.patch
        29 kB
        Phabricator
      7. ASF.LICENSE.NOT.GRANTED--D2775.5.patch
        28 kB
        Phabricator
      8. HBASE-5776-trunk.patch
        26 kB
        binlijin
      9. HBASE-5776-trunk-V2.patch
        27 kB
        binlijin
      10. 5776-trunk-V3.patch
        27 kB
        Ted Yu
      11. 5776-trunk-V4.patch
        27 kB
        Ted Yu
      12. 5776-trunk-V5.patch
        27 kB
        Ted Yu
      13. 5776-trunk-V6.patch
        27 kB
        Ted Yu
      14. 5776-trunk-V7.patch
        26 kB
        Ted Yu

        Activity

        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Liyin requested code review of "[jira][89-fb]HBASE-5776 HTableMultiplexer".
        Reviewers: Kannan

        There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.

        To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
        The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.

        Currently the HTableMultiplexer only supports the put operation.

        TEST PLAN
        Unit tests

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/client/HConnection.java
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        src/main/java/org/apache/hadoop/hbase/client/HTable.java
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/6303/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Liyin requested code review of " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Reviewers: Kannan There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. TEST PLAN Unit tests REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6303/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Liyin requested code review of "[jira][89-fb]HBASE-5776 HTableMultiplexer".
        Reviewers: Kannan

        There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster.

        To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation.
        The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server.

        Currently the HTableMultiplexer only supports the put operation.

        TEST PLAN
        Unit tests

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/client/HConnection.java
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        src/main/java/org/apache/hadoop/hbase/client/HTable.java
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/6303/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Liyin requested code review of " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Reviewers: Kannan There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. TEST PLAN Unit tests REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6303/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        tedyu has commented on the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".

        INLINE COMMENTS
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method.
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name.
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' -> 'and there is no successful put'
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared -> Prepare
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and'
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote.
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760.
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' -> caught
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk.
        src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ?
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount -> successfulCount
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ?
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ?
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping.
        Should we introduce some counter for the above case?
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please.

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - tedyu has commented on the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' -> 'and there is no successful put' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared -> Prepare src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' -> caught src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk. src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount -> successfulCount src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199 src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping. Should we introduce some counter for the above case? src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please. REVISION DETAIL https://reviews.facebook.net/D2775
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        tedyu has commented on the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".

        INLINE COMMENTS
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method.
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name.
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' -> 'and there is no successful put'
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared -> Prepare
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and'
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote.
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760.
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' -> caught
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk.
        src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ?
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount -> successfulCount
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ?
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ?
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping.
        Should we introduce some counter for the above case?
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please.

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - tedyu has commented on the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1701 processSingleMultiPut would be a better name for this method. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1720 If this log is to be kept, please add method name. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1727 'and there is no succeeded put' -> 'and there is no successful put' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1730 Prepared -> Prepare src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1702 Name this variable successfulPutCounter src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1738 Remove 'Set retry as true and' src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1745 Add a space between , and closing double quote. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1754 This is inconsistent with the handling on line 1760. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1766 'Catch the' -> caught src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:1 License. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:20 javadoc for class, please. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:25 HServerAddress is deprecated in trunk. src/main/java/org/apache/hadoop/hbase/client/HTable.java:714 Package private access should be enough. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:63 A 'continue' after this line would allow you to omit else line below. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:66 loc.getHostnamePort() would allow you to identify the server. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 This can be package private since it is used by test only. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:122 flush worker is per region server, do we need poolID here ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:172 succeededCount -> successfulCount src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:160 Give tmp a better name: how about putsUnderProcessing ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:198 ' put ' should precede 'requests' on line 199 src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 Should InterruptedException be handled specially ? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:208 If this worker gets busy, there may not be sleeping. Should we introduce some counter for the above case? src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:1 License, please. REVISION DETAIL https://reviews.facebook.net/D2775
        Hide
        tlipcon Todd Lipcon added a comment -

        Can this be done without adding a new interface? eg can we make the existing queueing/flushing behavior of HTable use a threadpool?

        Show
        tlipcon Todd Lipcon added a comment - Can this be done without adding a new interface? eg can we make the existing queueing/flushing behavior of HTable use a threadpool?
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Liyin updated the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".
        Reviewers: Kannan

        As discussed with Kannan offline, HTableMultiplexer will keep a pool htable instance. So user doesn't need to pass the htable as parameter for the put request. Also keep each put retrying several times before failing and clear the cache location after failing.

        Add more java doc and license.

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/client/HConnection.java
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        src/main/java/org/apache/hadoop/hbase/client/HTable.java
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Liyin updated the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Reviewers: Kannan As discussed with Kannan offline, HTableMultiplexer will keep a pool htable instance. So user doesn't need to pass the htable as parameter for the put request. Also keep each put retrying several times before failing and clear the cache location after failing. Add more java doc and license. REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Liyin updated the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".
        Reviewers: Kannan

        As discussed with Kannan offline, HTableMultiplexer will keep a pool htable instance. So user doesn't need to pass the htable as parameter for the put request. Also keep each put retrying several times before failing and clear the cache location after failing.

        Add more java doc and license.

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/client/HConnection.java
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        src/main/java/org/apache/hadoop/hbase/client/HTable.java
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Liyin updated the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Reviewers: Kannan As discussed with Kannan offline, HTableMultiplexer will keep a pool htable instance. So user doesn't need to pass the htable as parameter for the put request. Also keep each put retrying several times before failing and clear the cache location after failing. Add more java doc and license. REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
        Hide
        liyin Liyin Tang added a comment -

        @Todd, The HTableMultiplexer is designed to process the puts requests across different tables.
        All the puts across the tables will be sharded into each different queues based on their destination region server. It will help to batch more puts for each region server before sending out the rpc request.

        Show
        liyin Liyin Tang added a comment - @Todd, The HTableMultiplexer is designed to process the puts requests across different tables. All the puts across the tables will be sharded into each different queues based on their destination region server. It will help to batch more puts for each region server before sending out the rpc request.
        Hide
        tlipcon Todd Lipcon added a comment -

        Gotcha. But is this a user-facing API? or would the HTables themselves write into an HTable multiplexer? It seems to me like this kind of behavior should happen automatically for any writes going into HBase, without expanding our API footprint. Is there some reason that that's impossible?

        Show
        tlipcon Todd Lipcon added a comment - Gotcha. But is this a user-facing API? or would the HTables themselves write into an HTable multiplexer? It seems to me like this kind of behavior should happen automatically for any writes going into HBase, without expanding our API footprint. Is there some reason that that's impossible?
        Hide
        kannanm Kannan Muthukkaruppan added a comment -

        Todd:

        This is API is user-facing.

        The HTables themselves aren't thread safe, so applications end up using a separate HTable per thread, and the per-RS batching happens only within the context of a single HTable request. The HTableMultiplexer was intended as a simple wrapper helper/utility that allows the app to get "deferred put" functionality, but with batching even the puts coming from different application threads in most effective manner. It also supports puts to multiple tables to be grouped in the same batch. Extending HTables which are not thread safe for this use case didn't make sense, and also painful from backward compat perspective. In this approach, a single instance of HTableMultiplexer can be use by all the app threads to shove puts, and let the multiplexer group commits per RS, and forward them along in batches. This non-blocking multiplexer API could get more sophisticated in future, e.g., supporting async gets (with callbacks and so on) and different config knobs (e.g., how much buffering memory to allow for all queues, retry/failure handling logic via callbacks, etc.). And maybe, in future, we can gradually deprecate the deferred put functionality in HTable, and recommend all the non-blocking use cases to go via the multiplexer.

        Show
        kannanm Kannan Muthukkaruppan added a comment - Todd: This is API is user-facing. The HTables themselves aren't thread safe, so applications end up using a separate HTable per thread, and the per-RS batching happens only within the context of a single HTable request. The HTableMultiplexer was intended as a simple wrapper helper/utility that allows the app to get "deferred put" functionality, but with batching even the puts coming from different application threads in most effective manner. It also supports puts to multiple tables to be grouped in the same batch. Extending HTables which are not thread safe for this use case didn't make sense, and also painful from backward compat perspective. In this approach, a single instance of HTableMultiplexer can be use by all the app threads to shove puts, and let the multiplexer group commits per RS, and forward them along in batches. This non-blocking multiplexer API could get more sophisticated in future, e.g., supporting async gets (with callbacks and so on) and different config knobs (e.g., how much buffering memory to allow for all queues, retry/failure handling logic via callbacks, etc.). And maybe, in future, we can gradually deprecate the deferred put functionality in HTable, and recommend all the non-blocking use cases to go via the multiplexer.
        Hide
        zhihyu@ebaysf.com Ted Yu added a comment -

        Thanks for the explanation, Kannan.

        Since the Multiplexer isn't tied to any single table and it may support get's in the future, shall we remove the 'Table' in the class name ?

        Show
        zhihyu@ebaysf.com Ted Yu added a comment - Thanks for the explanation, Kannan. Since the Multiplexer isn't tied to any single table and it may support get's in the future, shall we remove the 'Table' in the class name ?
        Hide
        kannanm Kannan Muthukkaruppan added a comment -

        Ted: I think keeping "Table" in the name does not take way from the fact that it can support multiple tables. The multiplexer supports table operations-- so imo the name is ok as is.

        Show
        kannanm Kannan Muthukkaruppan added a comment - Ted: I think keeping "Table" in the name does not take way from the fact that it can support multiple tables. The multiplexer supports table operations-- so imo the name is ok as is.
        Hide
        otis Otis Gospodnetic added a comment -

        What happens when some of the puts fail even after N attempts? Does the caller get notified that a failure happened and which puts failed? If not, how should one deal with such situations?

        What happens with puts that are in memory, not yet written to RS, and the app dies/stops for whatever reason. Are those puts lost?

        Show
        otis Otis Gospodnetic added a comment - What happens when some of the puts fail even after N attempts? Does the caller get notified that a failure happened and which puts failed? If not, how should one deal with such situations? What happens with puts that are in memory, not yet written to RS, and the app dies/stops for whatever reason. Are those puts lost?
        Hide
        liyin Liyin Tang added a comment -

        @Otis,
        Good questions
        The motivation of HTableMultipler is NOT to replace the existing HTable but to provide a new alternative solutions for some high-throughput realtime applications. Usually, these applications can tolerate some data-loss during the region failover or other region-unavailable time instead of caching them in memory.

        In these cases, HTableMultiplexer tries its best to ONLY drop the data for these unavailable regions, instead of slowing down the overall multiput throughput which may cause more data dropping since the client fails to buffer any more incoming data across all the regions.

        Also, there is no call-back function if the put is finally failed after N attempts. And if client uses the multiput api in HTable, there is no such call back as well. However, we could have a way to config the HTableMultiplex to intelligently call back for some put operations.

        Show
        liyin Liyin Tang added a comment - @Otis, Good questions The motivation of HTableMultipler is NOT to replace the existing HTable but to provide a new alternative solutions for some high-throughput realtime applications. Usually, these applications can tolerate some data-loss during the region failover or other region-unavailable time instead of caching them in memory. In these cases, HTableMultiplexer tries its best to ONLY drop the data for these unavailable regions, instead of slowing down the overall multiput throughput which may cause more data dropping since the client fails to buffer any more incoming data across all the regions. Also, there is no call-back function if the put is finally failed after N attempts. And if client uses the multiput api in HTable, there is no such call back as well. However, we could have a way to config the HTableMultiplex to intelligently call back for some put operations.
        Hide
        otis Otis Gospodnetic added a comment -

        I read up on asynchbase yesterday. Doesn't asynchbase already solve the problem this issue is aimed at?
        See:
        http://search-hadoop.com/m/J6olJ11Idb
        http://search-hadoop.com/m/4fogb27wKWC

        Show
        otis Otis Gospodnetic added a comment - I read up on asynchbase yesterday. Doesn't asynchbase already solve the problem this issue is aimed at? See: http://search-hadoop.com/m/J6olJ11Idb http://search-hadoop.com/m/4fogb27wKWC
        Hide
        liyin Liyin Tang added a comment -

        Interesting Thanks @Otis for pointing the asynchbase out and I didn't know that before.

        I haven't read the asynchbase closely and it looks like both HTableMultiplexer and asynchbase are attempting to provide async (deferred) api for applications.
        However, I believe the HTableMulplixer mainly focuses on solving the slow region server problem by partition the put requests based on its destination region server before deferring it . So slow region server will only hammer one shard of the requests.

        BTW, is the asynchbase compatible with the current HBASE rpc protocol ?

        Show
        liyin Liyin Tang added a comment - Interesting Thanks @Otis for pointing the asynchbase out and I didn't know that before. I haven't read the asynchbase closely and it looks like both HTableMultiplexer and asynchbase are attempting to provide async (deferred) api for applications. However, I believe the HTableMulplixer mainly focuses on solving the slow region server problem by partition the put requests based on its destination region server before deferring it . So slow region server will only hammer one shard of the requests. BTW, is the asynchbase compatible with the current HBASE rpc protocol ?
        Hide
        otis Otis Gospodnetic added a comment -

        I think asynchbase does the same thing - has a queue for each RS.
        Compatibility - don't know off hand, check its repo on github.

        Show
        otis Otis Gospodnetic added a comment - I think asynchbase does the same thing - has a queue for each RS. Compatibility - don't know off hand, check its repo on github.
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Kannan has commented on the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".

        Liyin---

        good stuff! initial round of comments on this change.

        INLINE COMMENTS
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1700 function name: plural -> singular?
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:42 shared the same destination -> which share the same destination
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 simplify both line 92 & 95 to:

        the list of puts which could not be queued

        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:262 is there a possibility of a Concurrent Modification Exception since this map is not declared a concurrent map?
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1700 documentation needed for this function.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:183 is this lookup safe, when another thread might be inserting into this map? Doesn't this need to be a concurrent hash map?
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 the name of the param should have "put" somewhere and the fact that it is a "per-region server" queue size. Also, add some doc for the param here...
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:362 elapse -> elapsed
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:353 document param to avoid confusion (such as if this is a time or count based frequency) and if time what the units are, etc.
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:345 seems like this value can sometimes be inconsistent if you this happens right in between line 375 & 376.
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:67 isSucceeded -> success
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:394 each failed puts -> each failed put
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:426 s/and there are/and
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:424 Processed put <N> requests --> Processed <N> put requests
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:387 Is there a way to avoid preparing this map? At the very least, we can be lazy about creating this map when a failure actually happens. If there are 0 failures, we don't use this map.

        What comparator will be used for the "Put" keys in this map?

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        To: Kannan, Liyin
        Cc: JIRA, tedyu

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Kannan has commented on the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Liyin--- good stuff! initial round of comments on this change. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1700 function name: plural -> singular? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:42 shared the same destination -> which share the same destination src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:95 simplify both line 92 & 95 to: the list of puts which could not be queued src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:262 is there a possibility of a Concurrent Modification Exception since this map is not declared a concurrent map? src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1700 documentation needed for this function. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:183 is this lookup safe, when another thread might be inserting into this map? Doesn't this need to be a concurrent hash map? src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:211 the name of the param should have "put" somewhere and the fact that it is a "per-region server" queue size. Also, add some doc for the param here... src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:362 elapse -> elapsed src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:353 document param to avoid confusion (such as if this is a time or count based frequency) and if time what the units are, etc. src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:345 seems like this value can sometimes be inconsistent if you this happens right in between line 375 & 376. src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java:67 isSucceeded -> success src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:394 each failed puts -> each failed put src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:426 s/and there are/and src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:424 Processed put <N> requests --> Processed <N> put requests src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:387 Is there a way to avoid preparing this map? At the very least, we can be lazy about creating this map when a failure actually happens. If there are 0 failures, we don't use this map. What comparator will be used for the "Put" keys in this map? REVISION DETAIL https://reviews.facebook.net/D2775 To: Kannan, Liyin Cc: JIRA, tedyu
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Liyin updated the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".
        Reviewers: Kannan

        Address Kannan's comments.

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/client/HConnection.java
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        src/main/java/org/apache/hadoop/hbase/client/HTable.java
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
        src/main/java/org/apache/hadoop/hbase/client/MultiplexablePut.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

        To: Kannan, Liyin
        Cc: JIRA, tedyu

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Liyin updated the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Reviewers: Kannan Address Kannan's comments. REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/client/MultiplexablePut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java To: Kannan, Liyin Cc: JIRA, tedyu
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Liyin updated the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".
        Reviewers: Kannan

        Addressed some offline comments from Kannan about improving the failed put processing.

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/client/HConnection.java
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        src/main/java/org/apache/hadoop/hbase/client/HTable.java
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

        To: Kannan, Liyin
        Cc: JIRA, tedyu

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Liyin updated the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Reviewers: Kannan Addressed some offline comments from Kannan about improving the failed put processing. REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java To: Kannan, Liyin Cc: JIRA, tedyu
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Kannan has accepted the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".

        looks good Liyin. Remaining are cosmetic comments, hence accepting!

        INLINE COMMENTS
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1731 space after "to"
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:174 failed is unused
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:159 failed is unused
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:359 "-1" -> add space between - and 1.

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        BRANCH
        HBASE-5776

        To: Kannan, Liyin
        Cc: JIRA, tedyu

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Kannan has accepted the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". looks good Liyin. Remaining are cosmetic comments, hence accepting! INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java:1731 space after "to" src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:174 failed is unused src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:159 failed is unused src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:359 "-1" -> add space between - and 1. REVISION DETAIL https://reviews.facebook.net/D2775 BRANCH HBASE-5776 To: Kannan, Liyin Cc: JIRA, tedyu
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Liyin updated the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".
        Reviewers: Kannan

        Addressed Kannan's comments

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        AFFECTED FILES
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/client/HConnection.java
        src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
        src/main/java/org/apache/hadoop/hbase/client/HTable.java
        src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

        To: Kannan, Liyin
        Cc: JIRA, tedyu

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Liyin updated the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Reviewers: Kannan Addressed Kannan's comments REVISION DETAIL https://reviews.facebook.net/D2775 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnection.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/client/HTable.java src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java src/main/java/org/apache/hadoop/hbase/client/MultiPut.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java To: Kannan, Liyin Cc: JIRA, tedyu
        Hide
        phabricator@reviews.facebook.net Phabricator added a comment -

        Liyin has closed the revision "[jira][89-fb]HBASE-5776 HTableMultiplexer".

        Close this 89-fb revision and will port to apache trunk soon.

        REVISION DETAIL
        https://reviews.facebook.net/D2775

        To: Kannan, Liyin
        Cc: JIRA, tedyu

        Show
        phabricator@reviews.facebook.net Phabricator added a comment - Liyin has closed the revision " [jira] [89-fb] HBASE-5776 HTableMultiplexer". Close this 89-fb revision and will port to apache trunk soon. REVISION DETAIL https://reviews.facebook.net/D2775 To: Kannan, Liyin Cc: JIRA, tedyu
        Hide
        otis Otis Gospodnetic added a comment -

        Liyin Liang Any plans/ETA for getting this in trunk?

        Show
        otis Otis Gospodnetic added a comment - Liyin Liang Any plans/ETA for getting this in trunk?
        Hide
        aoxiang binlijin added a comment -

        I have make a patch for the trunk

        Show
        aoxiang binlijin added a comment - I have make a patch for the trunk
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -
        +public class TestHTableMultiplexer {
        

        Add annotation for test size. Is this a large test ?

        +public class HTableMultiplexer {
        

        Add annotation for audience and stability.

        +          start = elapsed = System.currentTimeMillis();
        

        Use EnvironmentEdgeManager above (and in other places).

        After you attach patch v2, please press 'Submit Patch' so that Hadoop QA can run test suite.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - + public class TestHTableMultiplexer { Add annotation for test size. Is this a large test ? + public class HTableMultiplexer { Add annotation for audience and stability. + start = elapsed = System .currentTimeMillis(); Use EnvironmentEdgeManager above (and in other places). After you attach patch v2, please press 'Submit Patch' so that Hadoop QA can run test suite.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12562042/HBASE-5776-trunk-V2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 29 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestFromClientSide
        org.apache.hadoop.hbase.client.TestHTableMultiplexer

        -1 core zombie tests. There are zombie tests. See build logs for details.

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562042/HBASE-5776-trunk-V2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 29 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.client.TestHTableMultiplexer -1 core zombie tests . There are zombie tests. See build logs for details. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3647//console This message is automatically generated.
        Hide
        aoxiang binlijin added a comment -

        java.lang.NullPointerException
        at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:940)
        at org.apache.hadoop.hbase.client.TestHTableMultiplexer.testHTableMultiplexer(TestHTableMultiplexer.java:96)

        May be the put failed, I run the test local success.
        mvn test -P localTests -DHBasePatchProcess -Dtest=org.apache.hadoop.hbase.client.TestHTableMultiplexer

        Show
        aoxiang binlijin added a comment - java.lang.NullPointerException at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:940) at org.apache.hadoop.hbase.client.TestHTableMultiplexer.testHTableMultiplexer(TestHTableMultiplexer.java:96) May be the put failed, I run the test local success. mvn test -P localTests -DHBasePatchProcess -Dtest=org.apache.hadoop.hbase.client.TestHTableMultiplexer
        Hide
        aoxiang binlijin added a comment -

        Sorry,the put should be success, because there is no Exception, but get return no data, i don't known why.

        Show
        aoxiang binlijin added a comment - Sorry,the put should be success, because there is no Exception, but get return no data, i don't known why.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Patch v3 corrects a problem where TestHTableMultiplexer was referring to "hbase.htablemultiplexer.flush.frequency" but HTableMultiplexer is using "hbase.htablemultiplexer.flush.frequency.ms"

        I created a constant HTableMultiplexer.TABLE_MULTIPLEXER_FLUSH_FREQ_MS so that test code refers to the same config param.

        I looped TestHTableMultiplexer 3 times and they passed.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Patch v3 corrects a problem where TestHTableMultiplexer was referring to "hbase.htablemultiplexer.flush.frequency" but HTableMultiplexer is using "hbase.htablemultiplexer.flush.frequency.ms" I created a constant HTableMultiplexer.TABLE_MULTIPLEXER_FLUSH_FREQ_MS so that test code refers to the same config param. I looped TestHTableMultiplexer 3 times and they passed.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12562236/5776-trunk-V3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestMultiParallel
        org.apache.hadoop.hbase.client.TestHTableMultiplexer

        -1 core zombie tests. There are zombie tests. See build logs for details.

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562236/5776-trunk-V3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestMultiParallel org.apache.hadoop.hbase.client.TestHTableMultiplexer -1 core zombie tests . There are zombie tests. See build logs for details. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3670//console This message is automatically generated.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Patch v4 added TEST_UTIL.waitUntilAllRegionsAssigned() following TEST_UTIL.createTable() call.

        TestHTableMultiplexer passes 3 times locally.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Patch v4 added TEST_UTIL.waitUntilAllRegionsAssigned() following TEST_UTIL.createTable() call. TestHTableMultiplexer passes 3 times locally.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12562239/5776-trunk-V4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestHTableMultiplexer

        -1 core zombie tests. There are zombie tests. See build logs for details.

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562239/5776-trunk-V4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHTableMultiplexer -1 core zombie tests . There are zombie tests. See build logs for details. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3672//console This message is automatically generated.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Patch v5 adds log statement so that it is easier to read the test output

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Patch v5 adds log statement so that it is easier to read the test output
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12562245/5776-trunk-V5.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestHTableMultiplexer
        org.apache.hadoop.hbase.ipc.TestDelayedRpc
        org.apache.hadoop.hbase.TestZooKeeper

        -1 core zombie tests. There are zombie tests. See build logs for details.

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562245/5776-trunk-V5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHTableMultiplexer org.apache.hadoop.hbase.ipc.TestDelayedRpc org.apache.hadoop.hbase.TestZooKeeper -1 core zombie tests . There are zombie tests. See build logs for details. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3673//console This message is automatically generated.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        From https://builds.apache.org/job/PreCommit-HBASE-Build/3673/testReport/org.apache.hadoop.hbase.client/TestHTableMultiplexer/testHTableMultiplexer/:

        2012-12-22 20:46:24,273 INFO  [pool-1-thread-1] client.TestHTableMultiplexer(93): Flushed put for  @ iteration 1
        ...
        2012-12-22 20:46:24,383 DEBUG [HTableFlushWorker-asf011.sp2.ygridcore.net:35330-0] client.HTableMultiplexer$HTableFlushWorker(553): Processed 1 put requests for asf011.sp2.ygridcore.net:35330 and 0 failed, latency for this send: 208
        

        We can see that client of HTableMultiplexer didn't get correct indication that the put was flushed.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - From https://builds.apache.org/job/PreCommit-HBASE-Build/3673/testReport/org.apache.hadoop.hbase.client/TestHTableMultiplexer/testHTableMultiplexer/: 2012-12-22 20:46:24,273 INFO [pool-1-thread-1] client.TestHTableMultiplexer(93): Flushed put for @ iteration 1 ... 2012-12-22 20:46:24,383 DEBUG [HTableFlushWorker-asf011.sp2.ygridcore.net:35330-0] client.HTableMultiplexer$HTableFlushWorker(553): Processed 1 put requests for asf011.sp2.ygridcore.net:35330 and 0 failed, latency for this send: 208 We can see that client of HTableMultiplexer didn't get correct indication that the put was flushed.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Patch v6 replaces connection.processBatch() call because it is deprecated.

        [WARNING] Javadoc Warnings
        [WARNING] javadoc: warning - No source files for package org.apache.hadoop.hbase.io.hfile.bucket
        [WARNING] /Users/tyu/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:117: warning - @param argument "put" is not a parameter name.
        

        I don't know where the first warning about hfile.bucket came from.

        I have corrected the second warning.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Patch v6 replaces connection.processBatch() call because it is deprecated. [WARNING] Javadoc Warnings [WARNING] javadoc: warning - No source files for package org.apache.hadoop.hbase.io.hfile.bucket [WARNING] /Users/tyu/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java:117: warning - @param argument "put" is not a parameter name. I don't know where the first warning about hfile.bucket came from. I have corrected the second warning.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12562251/5776-trunk-V6.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        -1 core zombie tests. There are zombie tests. See build logs for details.

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562251/5776-trunk-V6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 2 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . -1 core zombie tests . There are zombie tests. See build logs for details. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3674//console This message is automatically generated.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        [WARNING] javadoc: warning - No source files for package org.apache.hadoop.hbase.io.hfile.bucket

        The above was due to residue from application of HBASE-7404 patch.

        No javadoc warning after I removed the bucket directory.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - [WARNING] javadoc: warning - No source files for package org.apache.hadoop.hbase.io.hfile.bucket The above was due to residue from application of HBASE-7404 patch. No javadoc warning after I removed the bucket directory.
        Hide
        aoxiang binlijin added a comment -

        @Ted, Thank you very much, I am +1 on the Patch v6.

        Show
        aoxiang binlijin added a comment - @Ted, Thank you very much, I am +1 on the Patch v6.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment - - edited

        Will integrate patch v6 if there is no further review.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - - edited Will integrate patch v6 if there is no further review.
        Hide
        ram_krish ramkrishna.s.vasudevan added a comment -

        binlijin
        Did you compare this with asynchbase? Still i have not gone thro the patch closely. Some of the above comments says this one and asynchbase are similar.

        Show
        ram_krish ramkrishna.s.vasudevan added a comment - binlijin Did you compare this with asynchbase? Still i have not gone thro the patch closely. Some of the above comments says this one and asynchbase are similar.
        Hide
        aoxiang binlijin added a comment -

        @ramkrishna.s.vasudevan,no,i do not compare this with asynchbase,but this feature is simple compare with asynchbase.

        Show
        aoxiang binlijin added a comment - @ramkrishna.s.vasudevan,no,i do not compare this with asynchbase,but this feature is simple compare with asynchbase.
        Hide
        ram_krish ramkrishna.s.vasudevan added a comment -
         queue = serverToBufferQueueMap.get(addr);
            if (queue == null) {
              // Create the queue for the new region server
              queue = addNewRegionServer(addr, htable);
            }
        

        The above piece of code is in getBufferedQueue(). May be we can directly call addNewRegionServer() only. Also addNewRegionServer is also synchronized.

        Show
        ram_krish ramkrishna.s.vasudevan added a comment - queue = serverToBufferQueueMap.get(addr); if (queue == null ) { // Create the queue for the new region server queue = addNewRegionServer(addr, htable); } The above piece of code is in getBufferedQueue(). May be we can directly call addNewRegionServer() only. Also addNewRegionServer is also synchronized.
        Hide
        aoxiang binlijin added a comment -

        @ramkrishna.s.vasudevan,the getBufferedQueue is not synchronized,so we can get the buffer queue fast and put the put request into the queue fast. The addNewRegionServer is synchronized, so if we switch to it directly ,the cost will be huge.

        Show
        aoxiang binlijin added a comment - @ramkrishna.s.vasudevan,the getBufferedQueue is not synchronized,so we can get the buffer queue fast and put the put request into the queue fast. The addNewRegionServer is synchronized, so if we switch to it directly ,the cost will be huge.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Patch v7 addresses Ram's comment.

        TestHTableMultiplexer passes.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Patch v7 addresses Ram's comment. TestHTableMultiplexer passes.
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Looks like Hadoop QA was not running test suite against patch:
        https://builds.apache.org/job/PreCommit-HBASE-Build/3707/parameters/

        There is no effect on existing tests from patch v7 which is very close to patch v6 - there was no new modification to existing classes.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Looks like Hadoop QA was not running test suite against patch: https://builds.apache.org/job/PreCommit-HBASE-Build/3707/parameters/ There is no effect on existing tests from patch v7 which is very close to patch v6 - there was no new modification to existing classes.
        Hide
        aoxiang binlijin added a comment -

        @Ted, thanks , the patch is fine, +1 on the Patch

        Show
        aoxiang binlijin added a comment - @Ted, thanks , the patch is fine, +1 on the Patch
        Hide
        yuzhihong@gmail.com Ted Yu added a comment -

        Integrated to trunk.

        Thanks for the patch, Liyin and lijin.

        Thanks for the review, Ram.

        Show
        yuzhihong@gmail.com Ted Yu added a comment - Integrated to trunk. Thanks for the patch, Liyin and lijin. Thanks for the review, Ram.
        Hide
        hudson Hudson added a comment -

        Integrated in HBase-TRUNK #3660 (See https://builds.apache.org/job/HBase-TRUNK/3660/)
        HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
        Show
        hudson Hudson added a comment - Integrated in HBase-TRUNK #3660 (See https://builds.apache.org/job/HBase-TRUNK/3660/ ) HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
        Hide
        hudson Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #315 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/315/)
        HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
        Show
        hudson Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #315 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/315/ ) HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java
        Hide
        stack stack added a comment -

        Marking closed.

        Show
        stack stack added a comment - Marking closed.

          People

          • Assignee:
            aoxiang binlijin
            Reporter:
            liyin Liyin Tang
          • Votes:
            3 Vote for this issue
            Watchers:
            24 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development