Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-16388

Prevent client threads being blocked by only one slow region server

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.4.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Add a new configuration, hbase.client.perserver.requests.threshold, to limit the max number of concurrent request to one region server. If the user still create new request after reaching the limit, client will throw ServerTooBusyException and do not send the request to the server. This is a client side feature and can prevent client's threads being blocked by one slow region server resulting in the availability of client is much lower than the availability of region servers.

      For completeness, here extract on new config from hbase-default.xml:

      Property: hbase.client.perserver.requests.threshold
      Default: 2147483647
      Description: The max number of concurrent pending requests for one server in all client threads (process level). Exceeding requests will be thrown ServerTooBusyException immediately to prevent user's threads being occupied and blocked by only one slow region server. If you use a fix number of threads to access HBase in a synchronous way, set this to a suitable value which is related to the number of threads will help you. See https://issues.apache.org/jira/browse/HBASE-16388 for details.
      Show
      Add a new configuration, hbase.client.perserver.requests.threshold, to limit the max number of concurrent request to one region server. If the user still create new request after reaching the limit, client will throw ServerTooBusyException and do not send the request to the server. This is a client side feature and can prevent client's threads being blocked by one slow region server resulting in the availability of client is much lower than the availability of region servers. For completeness, here extract on new config from hbase-default.xml: Property: hbase.client.perserver.requests.threshold Default: 2147483647 Description: The max number of concurrent pending requests for one server in all client threads (process level). Exceeding requests will be thrown ServerTooBusyException immediately to prevent user's threads being occupied and blocked by only one slow region server. If you use a fix number of threads to access HBase in a synchronous way, set this to a suitable value which is related to the number of threads will help you. See https://issues.apache.org/jira/browse/HBASE-16388 for details.

      Description

      It is a general use case for HBase's users that they have several threads/handlers in their service, and each handler has its own Table/HTable instance. Generally users think each handler is independent and won't interact each other.

      However, in an extreme case, if a region server is very slow, every requests to this RS will timeout, handlers of users' service may be occupied by the long-waiting requests even requests belong to other RS will also be timeout.

      For example:
      If we have 100 handlers in a client service(timeout is 1000ms) and HBase has 10 region servers whose average response time is 50ms. If no region server is slow, we can handle 2000 requests per second.
      Now this service's QPS is 1000. If there is one region server very slow and all requests to it will be timeout. Users hope that only 10% requests failed, and 90% requests' response time is still 50ms, because only 10% requests are located to the slow RS. However, each second we have 100 long-waiting requests which exactly occupies all 100 handles. So all handlers is blocked, the availability of this service is almost zero.

      To prevent this case, we can limit the max concurrent requests to one RS in process-level. Requests exceeding the limit will throws ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above case, if we set this limit to 20, only 20 handlers will be occupied and other 80 handlers can still handle requests to other RS. The availability of this service is 90% as expected.

      1. HBASE-16388-branch-1-v1.patch
        15 kB
        Phil Yang
      2. HBASE-16388-branch-1-v2.patch
        15 kB
        Phil Yang
      3. HBASE-16388-v1.patch
        15 kB
        Phil Yang
      4. HBASE-16388-v2.patch
        15 kB
        Phil Yang
      5. HBASE-16388-v2.patch
        15 kB
        Phil Yang
      6. HBASE-16388-v2.patch
        15 kB
        Phil Yang
      7. HBASE-16388-v2.patch
        15 kB
        Phil Yang
      8. HBASE-16388-v3.patch
        15 kB
        Phil Yang

        Activity

        Hide
        yangzhe1991 Phil Yang added a comment -

        I'll upload a patch this week which adds a new configure, hbase.client.perserver.requests.threshold, to limit the max concurrent requests to one RS in process-level. Comments are welcomed, thanks.

        Show
        yangzhe1991 Phil Yang added a comment - I'll upload a patch this week which adds a new configure, hbase.client.perserver.requests.threshold, to limit the max concurrent requests to one RS in process-level. Comments are welcomed, thanks.
        Hide
        stack stack added a comment -

        Tell us more Phil Yang Where will you throttle the requests in the process? Hopefully it is not more on AsyncProcess (smile). But sounds great.

        Related, an interesting suggestion I heard last week was to do a queue per client on the regionserver. This would help when a 'bad client' tends to hog or swap the RegionServer... When the 'bad client' fills its queue, we can reject further requests. Meantime, other clients can make progress.

        Show
        stack stack added a comment - Tell us more Phil Yang Where will you throttle the requests in the process? Hopefully it is not more on AsyncProcess (smile). But sounds great. Related, an interesting suggestion I heard last week was to do a queue per client on the regionserver. This would help when a 'bad client' tends to hog or swap the RegionServer... When the 'bad client' fills its queue, we can reject further requests. Meantime, other clients can make progress.
        Hide
        aoxiang binlijin added a comment -

        May be we can using NettyRpcServer with it more easy by just adding a throttle Handler? Use netty we can control write to client and read from client.

        http://normanmaurer.me/presentations/2014-facebook-eng-netty/slides.html#9.0

        http://normanmaurer.me/presentations/2014-facebook-eng-netty/slides.html#33.0

        Show
        aoxiang binlijin added a comment - May be we can using NettyRpcServer with it more easy by just adding a throttle Handler? Use netty we can control write to client and read from client. http://normanmaurer.me/presentations/2014-facebook-eng-netty/slides.html#9.0 http://normanmaurer.me/presentations/2014-facebook-eng-netty/slides.html#33.0
        Hide
        yangzhe1991 Phil Yang added a comment -

        stack Thanks for your reply.

        Where will you throttle the requests in the process?

        In a lower level, tend to AbstractRpcClient. Use a static guava cache(it helps us cleaning up removed RS) to save AtomicLongs for each server address. One question is whether we need differentiate RPC requests to master and to RS, whether we need differentiate requests to meta table and user table. If not, the logic will be simple. And I think the concurrent requests to master or meta table is not very high, so a suitable threshold may not fail requests to master server or meta table, right?

        do a queue per client on the regionserver

        Sounds reasonable, it can prevent server being "DDOS" by a client. And in this issue we can prevent a client being blocked by a slow server. They are two approaches that both can increase the availability in extreme case.

        Show
        yangzhe1991 Phil Yang added a comment - stack Thanks for your reply. Where will you throttle the requests in the process? In a lower level, tend to AbstractRpcClient. Use a static guava cache(it helps us cleaning up removed RS) to save AtomicLongs for each server address. One question is whether we need differentiate RPC requests to master and to RS, whether we need differentiate requests to meta table and user table. If not, the logic will be simple. And I think the concurrent requests to master or meta table is not very high, so a suitable threshold may not fail requests to master server or meta table, right? do a queue per client on the regionserver Sounds reasonable, it can prevent server being "DDOS" by a client. And in this issue we can prevent a client being blocked by a slow server. They are two approaches that both can increase the availability in extreme case.
        Hide
        stack stack added a comment -

        That might work binlijin. We already have netty on client-side (Jurriaan Mous?)

        Show
        stack stack added a comment - That might work binlijin . We already have netty on client-side ( Jurriaan Mous ?)
        Hide
        stack stack added a comment -

        Thank you Phil Yang Start simple I'd say. Will be hard enough to test. Would be sweet if we could avoid another layer of accounting in the client. There is a bunch already going on in AsyncProcess by region and by server. Can this be leveraged?

        I like the way you characterize how queue-per-client is at other end of the spectrum from what this patch is trying to do. Looking forward to it Phil Yang.

        Show
        stack stack added a comment - Thank you Phil Yang Start simple I'd say. Will be hard enough to test. Would be sweet if we could avoid another layer of accounting in the client. There is a bunch already going on in AsyncProcess by region and by server. Can this be leveraged? I like the way you characterize how queue-per-client is at other end of the spectrum from what this patch is trying to do. Looking forward to it Phil Yang.
        Hide
        yangzhe1991 Phil Yang added a comment -

        Upload a patch for master. Logic is very simple. In the test case we should test Put alone because it use AsyncProcess which will wrap exceptions. And change the description of the confs used by AsyncProcess. hbase.client.max.*.tasks only affect mutation operations, not all operations in HTable.

        Show
        yangzhe1991 Phil Yang added a comment - Upload a patch for master. Logic is very simple. In the test case we should test Put alone because it use AsyncProcess which will wrap exceptions. And change the description of the confs used by AsyncProcess. hbase.client.max.*.tasks only affect mutation operations, not all operations in HTable.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 49s Maven dependency ordering for branch
        +1 mvninstall 5m 19s master passed
        +1 compile 1m 4s master passed with JDK v1.8.0_101
        +1 compile 1m 11s master passed with JDK v1.7.0_101
        +1 checkstyle 1m 22s master passed
        +1 mvneclipse 0m 54s master passed
        +1 findbugs 3m 38s master passed
        +1 javadoc 1m 9s master passed with JDK v1.8.0_101
        +1 javadoc 1m 13s master passed with JDK v1.7.0_101
        0 mvndep 0m 12s Maven dependency ordering for patch
        +1 mvninstall 1m 25s the patch passed
        +1 compile 1m 2s the patch passed with JDK v1.8.0_101
        +1 javac 1m 2s the patch passed
        +1 compile 1m 12s the patch passed with JDK v1.7.0_101
        +1 javac 1m 12s the patch passed
        +1 checkstyle 1m 7s the patch passed
        +1 mvneclipse 0m 40s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 hadoopcheck 29m 37s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
        +1 hbaseprotoc 0m 39s the patch passed
        +1 findbugs 4m 20s the patch passed
        +1 javadoc 1m 0s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 15s the patch passed with JDK v1.7.0_101
        +1 unit 1m 42s hbase-common in the patch passed.
        -1 unit 1m 1s hbase-client in the patch failed.
        -1 unit 113m 25s hbase-server in the patch failed.
        +1 asflicense 0m 54s The patch does not generate ASF License warnings.
        177m 38s



        Reason Tests
        Failed junit tests hadoop.hbase.TestInterfaceAudienceAnnotations
          hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
        Timed out junit tests org.apache.hadoop.hbase.TestNamespace



        Subsystem Report/Notes
        Docker Client=1.12.0 Server=1.12.0 Image:yetus/hbase:date2016-08-11
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823195/HBASE-16388-v1.patch
        JIRA Issue HBASE-16388
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml
        uname Linux 8ab4cfc19ebf 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / fb668a0
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/3064/artifact/patchprocess/patch-unit-hbase-client.txt
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/3064/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/3064/artifact/patchprocess/patch-unit-hbase-client.txt https://builds.apache.org/job/PreCommit-HBASE-Build/3064/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3064/testReport/
        modules C: hbase-common hbase-client hbase-server U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3064/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 49s Maven dependency ordering for branch +1 mvninstall 5m 19s master passed +1 compile 1m 4s master passed with JDK v1.8.0_101 +1 compile 1m 11s master passed with JDK v1.7.0_101 +1 checkstyle 1m 22s master passed +1 mvneclipse 0m 54s master passed +1 findbugs 3m 38s master passed +1 javadoc 1m 9s master passed with JDK v1.8.0_101 +1 javadoc 1m 13s master passed with JDK v1.7.0_101 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 1m 2s the patch passed with JDK v1.8.0_101 +1 javac 1m 2s the patch passed +1 compile 1m 12s the patch passed with JDK v1.7.0_101 +1 javac 1m 12s the patch passed +1 checkstyle 1m 7s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 hadoopcheck 29m 37s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. +1 hbaseprotoc 0m 39s the patch passed +1 findbugs 4m 20s the patch passed +1 javadoc 1m 0s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 15s the patch passed with JDK v1.7.0_101 +1 unit 1m 42s hbase-common in the patch passed. -1 unit 1m 1s hbase-client in the patch failed. -1 unit 113m 25s hbase-server in the patch failed. +1 asflicense 0m 54s The patch does not generate ASF License warnings. 177m 38s Reason Tests Failed junit tests hadoop.hbase.TestInterfaceAudienceAnnotations   hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures Timed out junit tests org.apache.hadoop.hbase.TestNamespace Subsystem Report/Notes Docker Client=1.12.0 Server=1.12.0 Image:yetus/hbase:date2016-08-11 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823195/HBASE-16388-v1.patch JIRA Issue HBASE-16388 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml uname Linux 8ab4cfc19ebf 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / fb668a0 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HBASE-Build/3064/artifact/patchprocess/patch-unit-hbase-client.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/3064/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/3064/artifact/patchprocess/patch-unit-hbase-client.txt https://builds.apache.org/job/PreCommit-HBASE-Build/3064/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3064/testReport/ modules C: hbase-common hbase-client hbase-server U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3064/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        allan163 Allan Yang added a comment - - edited

        what if there is only one region server serving? or what if the region on the regionserver is not balanced, some region server have a lot more regions than others. So threads to this server will naturally a lot more than others.

        Show
        allan163 Allan Yang added a comment - - edited what if there is only one region server serving? or what if the region on the regionserver is not balanced, some region server have a lot more regions than others. So threads to this server will naturally a lot more than others.
        Hide
        yangzhe1991 Phil Yang added a comment -

        It is configurable (default is 2147483647 which means disabled) and configuring the threshold should consider you concerns. The main objective is prevent slow region server blocking all client threads, so I think this threshold should be much higher than average number of threads accessing to one server, and is mainly depend on the number of threads in client.

        For example, if we have 100 threads in client, and we have 5 region servers. The threshold should be much higher than 20 to prevent failing on unbalanced-but-working servers, we can set the threshold to 50, so if a server is slow, we still have half of threads working well. If we have only one server, maybe this threshold is meaningless and we have to disable it.

        Show
        yangzhe1991 Phil Yang added a comment - It is configurable (default is 2147483647 which means disabled) and configuring the threshold should consider you concerns. The main objective is prevent slow region server blocking all client threads, so I think this threshold should be much higher than average number of threads accessing to one server, and is mainly depend on the number of threads in client. For example, if we have 100 threads in client, and we have 5 region servers. The threshold should be much higher than 20 to prevent failing on unbalanced-but-working servers, we can set the threshold to 50, so if a server is slow, we still have half of threads working well. If we have only one server, maybe this threshold is meaningless and we have to disable it.
        Hide
        yangzhe1991 Phil Yang added a comment -

        A further discussion:
        The best way to prevent threads of client being blocked by operations of Table is to support async HBase client. After HBASE-13784, users can use Staged Event-Driven Architecture to solve this more gracefully. If user must use blocking mode, this issue can avoid extreme case.

        Show
        yangzhe1991 Phil Yang added a comment - A further discussion: The best way to prevent threads of client being blocked by operations of Table is to support async HBase client. After HBASE-13784 , users can use Staged Event-Driven Architecture to solve this more gracefully. If user must use blocking mode, this issue can avoid extreme case.
        Hide
        yangzhe1991 Phil Yang added a comment -

        Add @InterfaceStability.Evolving for ServerBusyException

        Show
        yangzhe1991 Phil Yang added a comment - Add @InterfaceStability.Evolving for ServerBusyException
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 13m 14s Docker failed to build yetus/hbase:date2016-08-16.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823847/HBASE-16388-v2.patch
        JIRA Issue HBASE-16388
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3102/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 13m 14s Docker failed to build yetus/hbase:date2016-08-16. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823847/HBASE-16388-v2.patch JIRA Issue HBASE-16388 Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3102/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yangzhe1991 Phil Yang added a comment -

        retry

        Show
        yangzhe1991 Phil Yang added a comment - retry
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 18s Maven dependency ordering for branch
        +1 mvninstall 3m 23s master passed
        +1 compile 1m 24s master passed with JDK v1.8.0_101
        +1 compile 1m 16s master passed with JDK v1.7.0_101
        +1 checkstyle 1m 11s master passed
        +1 mvneclipse 0m 41s master passed
        +1 findbugs 3m 59s master passed
        +1 javadoc 1m 3s master passed with JDK v1.8.0_101
        +1 javadoc 1m 16s master passed with JDK v1.7.0_101
        0 mvndep 0m 12s Maven dependency ordering for patch
        +1 mvninstall 1m 27s the patch passed
        +1 compile 1m 5s the patch passed with JDK v1.8.0_101
        +1 javac 1m 5s the patch passed
        +1 compile 1m 13s the patch passed with JDK v1.7.0_101
        +1 javac 1m 13s the patch passed
        +1 checkstyle 1m 7s the patch passed
        +1 mvneclipse 0m 40s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 hadoopcheck 30m 36s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
        +1 hbaseprotoc 0m 39s the patch passed
        +1 findbugs 4m 45s the patch passed
        +1 javadoc 1m 6s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 17s the patch passed with JDK v1.7.0_101
        +1 unit 1m 48s hbase-common in the patch passed.
        +1 unit 1m 3s hbase-client in the patch passed.
        +1 unit 94m 14s hbase-server in the patch passed.
        +1 asflicense 0m 45s The patch does not generate ASF License warnings.
        157m 43s



        Subsystem Report/Notes
        Docker Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-17
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824069/HBASE-16388-v2.patch
        JIRA Issue HBASE-16388
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml
        uname Linux 1a6d1fc1fb60 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
        git revision master / d5080e8
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3118/testReport/
        modules C: hbase-common hbase-client hbase-server U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3118/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 3m 23s master passed +1 compile 1m 24s master passed with JDK v1.8.0_101 +1 compile 1m 16s master passed with JDK v1.7.0_101 +1 checkstyle 1m 11s master passed +1 mvneclipse 0m 41s master passed +1 findbugs 3m 59s master passed +1 javadoc 1m 3s master passed with JDK v1.8.0_101 +1 javadoc 1m 16s master passed with JDK v1.7.0_101 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 1m 5s the patch passed with JDK v1.8.0_101 +1 javac 1m 5s the patch passed +1 compile 1m 13s the patch passed with JDK v1.7.0_101 +1 javac 1m 13s the patch passed +1 checkstyle 1m 7s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 hadoopcheck 30m 36s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. +1 hbaseprotoc 0m 39s the patch passed +1 findbugs 4m 45s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 17s the patch passed with JDK v1.7.0_101 +1 unit 1m 48s hbase-common in the patch passed. +1 unit 1m 3s hbase-client in the patch passed. +1 unit 94m 14s hbase-server in the patch passed. +1 asflicense 0m 45s The patch does not generate ASF License warnings. 157m 43s Subsystem Report/Notes Docker Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-17 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824069/HBASE-16388-v2.patch JIRA Issue HBASE-16388 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml uname Linux 1a6d1fc1fb60 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh git revision master / d5080e8 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3118/testReport/ modules C: hbase-common hbase-client hbase-server U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3118/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        Apache9 Duo Zhang added a comment -

        How do you deal with the async call? Although it is not used right now and I plan to reimplement it...

        Show
        Apache9 Duo Zhang added a comment - How do you deal with the async call? Although it is not used right now and I plan to reimplement it...
        Hide
        yangzhe1991 Phil Yang added a comment -

        Will we use non-blocking Stub in blocking Table interface? This patch only handles BlockingStub. And I think AsyncTable won't block users' threads. So we need not have any limit of concurrent requests for a server in Stub if it is only used in AsyncTable.

        Show
        yangzhe1991 Phil Yang added a comment - Will we use non-blocking Stub in blocking Table interface? This patch only handles BlockingStub. And I think AsyncTable won't block users' threads. So we need not have any limit of concurrent requests for a server in Stub if it is only used in AsyncTable.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 25s Maven dependency ordering for branch
        +1 mvninstall 1m 56s branch-1 passed
        +1 compile 0m 55s branch-1 passed with JDK v1.8.0_101
        +1 compile 1m 6s branch-1 passed with JDK v1.7.0_101
        +1 checkstyle 1m 4s branch-1 passed
        +1 mvneclipse 0m 38s branch-1 passed
        -1 findbugs 1m 50s hbase-server in branch-1 has 1 extant Findbugs warnings.
        +1 javadoc 0m 52s branch-1 passed with JDK v1.8.0_101
        +1 javadoc 1m 8s branch-1 passed with JDK v1.7.0_101
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 19s the patch passed
        +1 compile 0m 57s the patch passed with JDK v1.8.0_101
        +1 javac 0m 57s the patch passed
        +1 compile 1m 6s the patch passed with JDK v1.7.0_101
        +1 javac 1m 6s the patch passed
        +1 checkstyle 1m 4s the patch passed
        +1 mvneclipse 0m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 hadoopcheck 16m 0s The patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
        +1 hbaseprotoc 0m 35s the patch passed
        +1 findbugs 4m 12s the patch passed
        +1 javadoc 0m 53s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 9s the patch passed with JDK v1.7.0_101
        +1 unit 1m 44s hbase-common in the patch passed.
        +1 unit 1m 35s hbase-client in the patch passed.
        -1 unit 83m 34s hbase-server in the patch failed.
        +1 asflicense 0m 43s The patch does not generate ASF License warnings.
        128m 25s



        Reason Tests
        Failed junit tests hadoop.hbase.regionserver.TestHRegion
          hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
        Timed out junit tests org.apache.hadoop.hbase.regionserver.TestClusterId



        Subsystem Report/Notes
        Docker Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-18
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824320/HBASE-16388-branch-1-v1.patch
        JIRA Issue HBASE-16388
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml
        uname Linux 7524f0f5d6c3 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/patchprocess/precommit/personality/hbase.sh
        git revision branch-1 / e05b002
        Default Java 1.7.0_101
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/3142/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/3142/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/3142/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3142/testReport/
        modules C: hbase-common hbase-client hbase-server U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3142/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 1m 56s branch-1 passed +1 compile 0m 55s branch-1 passed with JDK v1.8.0_101 +1 compile 1m 6s branch-1 passed with JDK v1.7.0_101 +1 checkstyle 1m 4s branch-1 passed +1 mvneclipse 0m 38s branch-1 passed -1 findbugs 1m 50s hbase-server in branch-1 has 1 extant Findbugs warnings. +1 javadoc 0m 52s branch-1 passed with JDK v1.8.0_101 +1 javadoc 1m 8s branch-1 passed with JDK v1.7.0_101 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 0m 57s the patch passed with JDK v1.8.0_101 +1 javac 0m 57s the patch passed +1 compile 1m 6s the patch passed with JDK v1.7.0_101 +1 javac 1m 6s the patch passed +1 checkstyle 1m 4s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 hadoopcheck 16m 0s The patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. +1 hbaseprotoc 0m 35s the patch passed +1 findbugs 4m 12s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 9s the patch passed with JDK v1.7.0_101 +1 unit 1m 44s hbase-common in the patch passed. +1 unit 1m 35s hbase-client in the patch passed. -1 unit 83m 34s hbase-server in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 128m 25s Reason Tests Failed junit tests hadoop.hbase.regionserver.TestHRegion   hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat Timed out junit tests org.apache.hadoop.hbase.regionserver.TestClusterId Subsystem Report/Notes Docker Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-18 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824320/HBASE-16388-branch-1-v1.patch JIRA Issue HBASE-16388 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml uname Linux 7524f0f5d6c3 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/hbase.sh git revision branch-1 / e05b002 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/3142/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html unit https://builds.apache.org/job/PreCommit-HBASE-Build/3142/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/3142/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3142/testReport/ modules C: hbase-common hbase-client hbase-server U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3142/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yangzhe1991 Phil Yang added a comment -

        Failed tests are unrelated and can pass locally

        Show
        yangzhe1991 Phil Yang added a comment - Failed tests are unrelated and can pass locally
        Hide
        yangzhe1991 Phil Yang added a comment -

        Any comments? Thanks

        Show
        yangzhe1991 Phil Yang added a comment - Any comments? Thanks
        Hide
        yangzhe1991 Phil Yang added a comment -

        Let's see if the patch still works for master

        Show
        yangzhe1991 Phil Yang added a comment - Let's see if the patch still works for master
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 6m 27s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 42s Maven dependency ordering for branch
        +1 mvninstall 4m 56s master passed
        +1 compile 1m 24s master passed with JDK v1.8.0_101
        +1 compile 1m 24s master passed with JDK v1.7.0_111
        +1 checkstyle 1m 16s master passed
        +1 mvneclipse 0m 47s master passed
        +1 findbugs 3m 54s master passed
        +1 javadoc 1m 13s master passed with JDK v1.8.0_101
        +1 javadoc 1m 27s master passed with JDK v1.7.0_111
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 1m 41s the patch passed
        +1 compile 1m 14s the patch passed with JDK v1.8.0_101
        +1 javac 1m 14s the patch passed
        +1 compile 1m 23s the patch passed with JDK v1.7.0_111
        +1 javac 1m 23s the patch passed
        +1 checkstyle 1m 13s the patch passed
        +1 mvneclipse 0m 44s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 hadoopcheck 36m 32s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
        +1 hbaseprotoc 0m 50s the patch passed
        +1 findbugs 4m 32s the patch passed
        +1 javadoc 1m 9s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 28s the patch passed with JDK v1.7.0_111
        +1 unit 1m 57s hbase-common in the patch passed.
        +1 unit 1m 8s hbase-client in the patch passed.
        -1 unit 124m 5s hbase-server in the patch failed.
        +1 asflicense 1m 15s The patch does not generate ASF License warnings.
        204m 27s



        Reason Tests
        Failed junit tests hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
          hadoop.hbase.client.TestAdmin1
          hadoop.hbase.regionserver.TestHRegion
        Timed out junit tests org.apache.hadoop.hbase.TestHBaseTestingUtility
          org.apache.hadoop.hbase.TestZooKeeper
          org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot
          org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
          org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot



        Subsystem Report/Notes
        Docker Client=1.12.1 Server=1.12.1 Image:yetus/hbase:date2016-09-08
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827503/HBASE-16388-v2.patch
        JIRA Issue HBASE-16388
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml
        uname Linux f78efc7df604 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / fc224ed
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/3452/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/3452/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3452/testReport/
        modules C: hbase-common hbase-client hbase-server U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3452/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 6m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 42s Maven dependency ordering for branch +1 mvninstall 4m 56s master passed +1 compile 1m 24s master passed with JDK v1.8.0_101 +1 compile 1m 24s master passed with JDK v1.7.0_111 +1 checkstyle 1m 16s master passed +1 mvneclipse 0m 47s master passed +1 findbugs 3m 54s master passed +1 javadoc 1m 13s master passed with JDK v1.8.0_101 +1 javadoc 1m 27s master passed with JDK v1.7.0_111 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 41s the patch passed +1 compile 1m 14s the patch passed with JDK v1.8.0_101 +1 javac 1m 14s the patch passed +1 compile 1m 23s the patch passed with JDK v1.7.0_111 +1 javac 1m 23s the patch passed +1 checkstyle 1m 13s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 hadoopcheck 36m 32s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. +1 hbaseprotoc 0m 50s the patch passed +1 findbugs 4m 32s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 28s the patch passed with JDK v1.7.0_111 +1 unit 1m 57s hbase-common in the patch passed. +1 unit 1m 8s hbase-client in the patch passed. -1 unit 124m 5s hbase-server in the patch failed. +1 asflicense 1m 15s The patch does not generate ASF License warnings. 204m 27s Reason Tests Failed junit tests hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures   hadoop.hbase.client.TestAdmin1   hadoop.hbase.regionserver.TestHRegion Timed out junit tests org.apache.hadoop.hbase.TestHBaseTestingUtility   org.apache.hadoop.hbase.TestZooKeeper   org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot   org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot   org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot Subsystem Report/Notes Docker Client=1.12.1 Server=1.12.1 Image:yetus/hbase:date2016-09-08 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827503/HBASE-16388-v2.patch JIRA Issue HBASE-16388 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml uname Linux f78efc7df604 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / fc224ed Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HBASE-Build/3452/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/3452/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3452/testReport/ modules C: hbase-common hbase-client hbase-server U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3452/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 3m 9s master passed
        +1 compile 0m 59s master passed with JDK v1.8.0_101
        +1 compile 1m 10s master passed with JDK v1.7.0_111
        +1 checkstyle 1m 7s master passed
        +1 mvneclipse 0m 38s master passed
        +1 findbugs 3m 39s master passed
        +1 javadoc 0m 54s master passed with JDK v1.8.0_101
        +1 javadoc 1m 13s master passed with JDK v1.7.0_111
        0 mvndep 0m 12s Maven dependency ordering for patch
        +1 mvninstall 1m 23s the patch passed
        +1 compile 0m 58s the patch passed with JDK v1.8.0_101
        +1 javac 0m 58s the patch passed
        +1 compile 1m 9s the patch passed with JDK v1.7.0_111
        +1 javac 1m 9s the patch passed
        +1 checkstyle 1m 8s the patch passed
        +1 mvneclipse 0m 38s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 hadoopcheck 27m 36s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
        +1 hbaseprotoc 0m 36s the patch passed
        +1 findbugs 4m 20s the patch passed
        +1 javadoc 0m 54s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 12s the patch passed with JDK v1.7.0_111
        +1 unit 1m 47s hbase-common in the patch passed.
        +1 unit 1m 0s hbase-client in the patch passed.
        +1 unit 91m 25s hbase-server in the patch passed.
        +1 asflicense 0m 51s The patch does not generate ASF License warnings.
        149m 22s



        Subsystem Report/Notes
        Docker Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-08
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827532/HBASE-16388-v2.patch
        JIRA Issue HBASE-16388
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml
        uname Linux c81dd854dba3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / fc224ed
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3458/testReport/
        modules C: hbase-common hbase-client hbase-server U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3458/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 3m 9s master passed +1 compile 0m 59s master passed with JDK v1.8.0_101 +1 compile 1m 10s master passed with JDK v1.7.0_111 +1 checkstyle 1m 7s master passed +1 mvneclipse 0m 38s master passed +1 findbugs 3m 39s master passed +1 javadoc 0m 54s master passed with JDK v1.8.0_101 +1 javadoc 1m 13s master passed with JDK v1.7.0_111 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 0m 58s the patch passed with JDK v1.8.0_101 +1 javac 0m 58s the patch passed +1 compile 1m 9s the patch passed with JDK v1.7.0_111 +1 javac 1m 9s the patch passed +1 checkstyle 1m 8s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 hadoopcheck 27m 36s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. +1 hbaseprotoc 0m 36s the patch passed +1 findbugs 4m 20s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 12s the patch passed with JDK v1.7.0_111 +1 unit 1m 47s hbase-common in the patch passed. +1 unit 1m 0s hbase-client in the patch passed. +1 unit 91m 25s hbase-server in the patch passed. +1 asflicense 0m 51s The patch does not generate ASF License warnings. 149m 22s Subsystem Report/Notes Docker Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-08 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827532/HBASE-16388-v2.patch JIRA Issue HBASE-16388 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml uname Linux c81dd854dba3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / fc224ed Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3458/testReport/ modules C: hbase-common hbase-client hbase-server U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3458/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yangzhe1991 Phil Yang added a comment -

        stack Any concerns? Thanks.

        Show
        yangzhe1991 Phil Yang added a comment - stack Any concerns? Thanks.
        Hide
        stack stack added a comment -

        Phil Yang Can this be on by default rather than require config to enable? Maybe on by default in master but in branch-1, you have to enable it? It is already 'complaining' loudly when up against the limit by throwing the ServerBusyException but maybe add that we are up against config limit in the message and config to set to up limit in the SBE message ... and that the SBE is coming from client-side rather than server (may not be immediately obvious)?

        Nice test.

        Needs a release note because this is important issue w/ nice fix.

        Thanks

        Show
        stack stack added a comment - Phil Yang Can this be on by default rather than require config to enable? Maybe on by default in master but in branch-1, you have to enable it? It is already 'complaining' loudly when up against the limit by throwing the ServerBusyException but maybe add that we are up against config limit in the message and config to set to up limit in the SBE message ... and that the SBE is coming from client-side rather than server (may not be immediately obvious)? Nice test. Needs a release note because this is important issue w/ nice fix. Thanks
        Hide
        yangzhe1991 Phil Yang added a comment -

        We limit the number of concurrent requests. The suitable conf depends on the number of RSs and the number of threads accessing Table's API. But we only know the number of RSs, we don't know the number of threads. So this should be set by users according to their own environment.

        SBE is from client-side, we don't send request at all. So this is a client-only fix. After we have AsyncTable and users use it in an async way which means user's threads will not be blocked, this conf is still useful to prevent we have too many pending request resulting in an OOM. And of course for AsyncTable's user, this limit can be much larger than blocking Table's user.

        Show
        yangzhe1991 Phil Yang added a comment - We limit the number of concurrent requests. The suitable conf depends on the number of RSs and the number of threads accessing Table's API. But we only know the number of RSs, we don't know the number of threads. So this should be set by users according to their own environment. SBE is from client-side, we don't send request at all. So this is a client-only fix. After we have AsyncTable and users use it in an async way which means user's threads will not be blocked, this conf is still useful to prevent we have too many pending request resulting in an OOM. And of course for AsyncTable's user, this limit can be much larger than blocking Table's user.
        Hide
        stack stack added a comment -

        Where does the user get feedback on how well or how bad their setting of this config is doing Phil Yang?

        On SBE, I confused it with RegionTooBusyException.java. Pardon me. I should have seen it is a new Exception. So ignore my remark. Answer above question and I'll commit (after changing name of SBE to ServerTooBusyException and adding detail on this new config.) Thanks Phil Yang

        Show
        stack stack added a comment - Where does the user get feedback on how well or how bad their setting of this config is doing Phil Yang ? On SBE, I confused it with RegionTooBusyException.java. Pardon me. I should have seen it is a new Exception. So ignore my remark. Answer above question and I'll commit (after changing name of SBE to ServerTooBusyException and adding detail on this new config.) Thanks Phil Yang
        Hide
        yangzhe1991 Phil Yang added a comment - - edited

        If a user connect to a cluster with N region servers, we expected that if one region server is slow and all requests to it timed out, at a higher level which can monitor the service using HBase client, there should be only 1/N requests failed/timeout. If the failed requests are more than 1/N, especially much more than 1/N, the limit should be change to a lower number.

        And if there is no slow server in the cluster, but clients throw SBE, there may be two reasons: the limit is too strict, or the regions are not balanced and much more than 1/N keys belong to the server.

        So the proper limit should be several times of 1/N*threadNumber, too high may reduce the availability when there are slow servers, too low may prevent normal requests. And a relatively balanced cluster is required.

        Show
        yangzhe1991 Phil Yang added a comment - - edited If a user connect to a cluster with N region servers, we expected that if one region server is slow and all requests to it timed out, at a higher level which can monitor the service using HBase client, there should be only 1/N requests failed/timeout. If the failed requests are more than 1/N, especially much more than 1/N, the limit should be change to a lower number. And if there is no slow server in the cluster, but clients throw SBE, there may be two reasons: the limit is too strict, or the regions are not balanced and much more than 1/N keys belong to the server. So the proper limit should be several times of 1/N*threadNumber, too high may reduce the availability when there are slow servers, too low may prevent normal requests. And a relatively balanced cluster is required.
        Hide
        stack stack added a comment -

        Ok. Expert config. If you are seeing SBE, then you have set hbase.client.perserver.requests.threshold down from its default value so you know about it.

        The patch has rotted because of HBASE-16445 "Refactor and reimplement RpcClient" . Please redo, change ServerBusyException to ServerTooBusyException just so it has same form as RegionTooBusyException and I'm +. Thanks Phil Yang

        Show
        stack stack added a comment - Ok. Expert config. If you are seeing SBE, then you have set hbase.client.perserver.requests.threshold down from its default value so you know about it. The patch has rotted because of HBASE-16445 "Refactor and reimplement RpcClient" . Please redo, change ServerBusyException to ServerTooBusyException just so it has same form as RegionTooBusyException and I'm +. Thanks Phil Yang
        Hide
        yangzhe1991 Phil Yang added a comment -

        Upload rebased patch for master

        Show
        yangzhe1991 Phil Yang added a comment - Upload rebased patch for master
        Hide
        yangzhe1991 Phil Yang added a comment -

        Patch for branch-1, changed the name of exception

        Show
        yangzhe1991 Phil Yang added a comment - Patch for branch-1, changed the name of exception
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 4m 6s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 5s Maven dependency ordering for branch
        +1 mvninstall 5m 58s branch-1 passed
        +1 compile 1m 7s branch-1 passed with JDK v1.8.0_101
        +1 compile 1m 12s branch-1 passed with JDK v1.7.0_111
        +1 checkstyle 1m 36s branch-1 passed
        +1 mvneclipse 0m 53s branch-1 passed
        -1 findbugs 2m 11s hbase-server in branch-1 has 1 extant Findbugs warnings.
        +1 javadoc 1m 6s branch-1 passed with JDK v1.8.0_101
        +1 javadoc 1m 19s branch-1 passed with JDK v1.7.0_111
        0 mvndep 0m 12s Maven dependency ordering for patch
        +1 mvninstall 1m 32s the patch passed
        +1 compile 1m 8s the patch passed with JDK v1.8.0_101
        +1 javac 1m 8s the patch passed
        +1 compile 1m 17s the patch passed with JDK v1.7.0_111
        +1 javac 1m 17s the patch passed
        +1 checkstyle 1m 11s the patch passed
        +1 mvneclipse 0m 42s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 hadoopcheck 20m 28s The patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
        +1 hbaseprotoc 0m 46s the patch passed
        +1 findbugs 4m 56s the patch passed
        +1 javadoc 1m 0s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 15s the patch passed with JDK v1.7.0_111
        +1 unit 1m 57s hbase-common in the patch passed.
        +1 unit 1m 56s hbase-client in the patch passed.
        -1 unit 93m 51s hbase-server in the patch failed.
        +1 asflicense 0m 51s The patch does not generate ASF License warnings.
        156m 54s



        Reason Tests
        Failed junit tests hadoop.hbase.regionserver.TestHRegion
          hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat
          hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat



        Subsystem Report/Notes
        Docker Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-14
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828394/HBASE-16388-branch-1-v2.patch
        JIRA Issue HBASE-16388
        Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml
        uname Linux bbb4bfa64741 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/patchprocess/precommit/personality/hbase.sh
        git revision branch-1 / ac1ee77
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/3545/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/3545/artifact/patchprocess/patch-unit-hbase-server.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/3545/artifact/patchprocess/patch-unit-hbase-server.txt
        Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3545/testReport/
        modules C: hbase-common hbase-client hbase-server U: .
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3545/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 4m 6s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 5s Maven dependency ordering for branch +1 mvninstall 5m 58s branch-1 passed +1 compile 1m 7s branch-1 passed with JDK v1.8.0_101 +1 compile 1m 12s branch-1 passed with JDK v1.7.0_111 +1 checkstyle 1m 36s branch-1 passed +1 mvneclipse 0m 53s branch-1 passed -1 findbugs 2m 11s hbase-server in branch-1 has 1 extant Findbugs warnings. +1 javadoc 1m 6s branch-1 passed with JDK v1.8.0_101 +1 javadoc 1m 19s branch-1 passed with JDK v1.7.0_111 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 32s the patch passed +1 compile 1m 8s the patch passed with JDK v1.8.0_101 +1 javac 1m 8s the patch passed +1 compile 1m 17s the patch passed with JDK v1.7.0_111 +1 javac 1m 17s the patch passed +1 checkstyle 1m 11s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 hadoopcheck 20m 28s The patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. +1 hbaseprotoc 0m 46s the patch passed +1 findbugs 4m 56s the patch passed +1 javadoc 1m 0s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 15s the patch passed with JDK v1.7.0_111 +1 unit 1m 57s hbase-common in the patch passed. +1 unit 1m 56s hbase-client in the patch passed. -1 unit 93m 51s hbase-server in the patch failed. +1 asflicense 0m 51s The patch does not generate ASF License warnings. 156m 54s Reason Tests Failed junit tests hadoop.hbase.regionserver.TestHRegion   hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat   hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat Subsystem Report/Notes Docker Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-14 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828394/HBASE-16388-branch-1-v2.patch JIRA Issue HBASE-16388 Optional Tests asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml uname Linux bbb4bfa64741 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/hbase.sh git revision branch-1 / ac1ee77 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HBASE-Build/3545/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html unit https://builds.apache.org/job/PreCommit-HBASE-Build/3545/artifact/patchprocess/patch-unit-hbase-server.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/3545/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/3545/testReport/ modules C: hbase-common hbase-client hbase-server U: . Console output https://builds.apache.org/job/PreCommit-HBASE-Build/3545/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        stack stack added a comment -

        Findbugs issue is unrelated:

        Code Warning
        RV Return value of java.util.concurrent.CountDownLatch.await(long, TimeUnit) ignored in org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper()

        Looks like attempt at fixing this elsewhere did not work.

        I tried the tests locally and they passed. Need to dig in on these tests on why they are failing. They seem unrelated to this patch. Going to commit.

        Show
        stack stack added a comment - Findbugs issue is unrelated: Code Warning RV Return value of java.util.concurrent.CountDownLatch.await(long, TimeUnit) ignored in org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper() Looks like attempt at fixing this elsewhere did not work. I tried the tests locally and they passed. Need to dig in on these tests on why they are failing. They seem unrelated to this patch. Going to commit.
        Hide
        stack stack added a comment -

        Pushed to branch-1 and master. Thanks for the nice patch Phil Yang I tried to backport to 1.3 but it wouldn't go back.

        Show
        stack stack added a comment - Pushed to branch-1 and master. Thanks for the nice patch Phil Yang I tried to backport to 1.3 but it wouldn't go back.
        Hide
        jinghe Jerry He added a comment -

        Good feature! Just need the release notes and the hbase-default.xml to be more clear. The threshold limit is per Connection/HConnection.
        This is to compare to a situation where multi-threaded client opens multiple HConnections,

        Show
        jinghe Jerry He added a comment - Good feature! Just need the release notes and the hbase-default.xml to be more clear. The threshold limit is per Connection/HConnection. This is to compare to a situation where multi-threaded client opens multiple HConnections,
        Hide
        yangzhe1991 Phil Yang added a comment -

        As said in hbase-default, the limit is process-level, even user uses multiple HConnections, because the counter cache is a static object.

        Show
        yangzhe1991 Phil Yang added a comment - As said in hbase-default, the limit is process-level, even user uses multiple HConnections, because the counter cache is a static object.
        Hide
        yangzhe1991 Phil Yang added a comment -

        Thanks for reviews and committing.

        I'm not familiar with the release strategy. I didn't make a patch for branch-1.3 because I thought we will not add new feature to 1.3. If we need this feature in 1.3 I can make a patch for 1.3 tomorrow(UTC+8)

        Show
        yangzhe1991 Phil Yang added a comment - Thanks for reviews and committing. I'm not familiar with the release strategy. I didn't make a patch for branch-1.3 because I thought we will not add new feature to 1.3. If we need this feature in 1.3 I can make a patch for 1.3 tomorrow(UTC+8)
        Hide
        stack stack added a comment -

        Added the hbase-default.xml extract to the release note (after Jerry He's questions above)

        Show
        stack stack added a comment - Added the hbase-default.xml extract to the release note (after Jerry He 's questions above)
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1601 (See https://builds.apache.org/job/HBase-Trunk_matrix/1601/)
        HBASE-16388 Prevent client threads being blocked by only one slow region (stack: rev 8ef6c76344127f2f4d2f9536d87fa6fc7b5c7132)

        • (edit) hbase-common/src/main/resources/hbase-default.xml
        • (add) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/ServerTooBusyException.java
        • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
        • (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
        • (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1601 (See https://builds.apache.org/job/HBase-Trunk_matrix/1601/ ) HBASE-16388 Prevent client threads being blocked by only one slow region (stack: rev 8ef6c76344127f2f4d2f9536d87fa6fc7b5c7132) (edit) hbase-common/src/main/resources/hbase-default.xml (add) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/ServerTooBusyException.java (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
        Hide
        jinghe Jerry He added a comment -

        because the counter cache is a static object.

        Got it, Thanks.

        Show
        jinghe Jerry He added a comment - because the counter cache is a static object. Got it, Thanks.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build HBase-1.4 #416 (See https://builds.apache.org/job/HBase-1.4/416/)
        HBASE-16388 Prevent client threads being blocked by only one slow region (stack: rev 069d1f73fa7e1a2b4c21ba95dea867d077a51068)

        • (add) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/ServerTooBusyException.java
        • (edit) hbase-common/src/main/resources/hbase-default.xml
        • (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
        • (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
        • (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-1.4 #416 (See https://builds.apache.org/job/HBase-1.4/416/ ) HBASE-16388 Prevent client threads being blocked by only one slow region (stack: rev 069d1f73fa7e1a2b4c21ba95dea867d077a51068) (add) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/ServerTooBusyException.java (edit) hbase-common/src/main/resources/hbase-default.xml (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
        Hide
        mantonov Mikhail Antonov added a comment -

        Phil Yang

        I good one, I missed that. Is it fair to say that the primary motivation for that is that global, per-region and per-server limits in AP are flawed since they only ever enforced on the write path (going through AP#submit() / buffered mutator)?

        With that being client-only change I'd consider backporting it to 1.3.. Anything that reduced blast radius from bad RS is an important reliability fix IMO.

        Show
        mantonov Mikhail Antonov added a comment - Phil Yang I good one, I missed that. Is it fair to say that the primary motivation for that is that global, per-region and per-server limits in AP are flawed since they only ever enforced on the write path (going through AP#submit() / buffered mutator)? With that being client-only change I'd consider backporting it to 1.3.. Anything that reduced blast radius from bad RS is an important reliability fix IMO.
        Hide
        yangzhe1991 Phil Yang added a comment -

        Yes, this is a client-only change covering all rpc requests while AP's limit only covering part of operations in HTable.

        Show
        yangzhe1991 Phil Yang added a comment - Yes, this is a client-only change covering all rpc requests while AP's limit only covering part of operations in HTable.
        Hide
        rayokota Robert Yokota added a comment -

        Using "hbase.client.perserver.requests.threshold" would be much easier if there were metrics for the value that is tested against the threshold. I've submitted HBASE-18559 with a possible way to do this, but perhaps there is a better way.

        Show
        rayokota Robert Yokota added a comment - Using "hbase.client.perserver.requests.threshold" would be much easier if there were metrics for the value that is tested against the threshold. I've submitted HBASE-18559 with a possible way to do this, but perhaps there is a better way.

          People

          • Assignee:
            yangzhe1991 Phil Yang
            Reporter:
            yangzhe1991 Phil Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            19 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development