Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11724

Add to RWQueueRpcExecutor the ability to split get and scan handlers

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 2.0.0
    • Component/s: IPC/RPC
    • Labels:
      None

      Description

      RWQueueRpcExecutor has the devision between reads and writes requests, but we can split also small-reads and long-reads. This can be useful to force a deprioritization of scans on the RS.

      1. HBASE-11724-v2.patch
        19 kB
        Matteo Bertozzi
      2. HBASE-11724-v1.patch
        19 kB
        Matteo Bertozzi
      3. HBASE-11724-v0.patch
        16 kB
        Matteo Bertozzi

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12661226/HBASE-11724-v0.patch
          against trunk revision .
          ATTACHMENT ID: 12661226

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestTableLockManager

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12661226/HBASE-11724-v0.patch against trunk revision . ATTACHMENT ID: 12661226 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestTableLockManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10401//console This message is automatically generated.
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          Without reading the code, this is ambiguous - what does 0 or 1 specifically mean? what is the default setting? does 0 mean get+scan shared, and 1 mean separate? is it a float? (it is) I it a sub share of h.i.s.callqueue.read.share or is it at the "same level"? (it is not).

          (related question - is hbase.ipc.server.callqueue.read.share a float or is it specifically 0, 0.5 and 1.0 values?)

          +    <name>hbase.ipc.server.callqueue.scan.share</name>
          +    <value>0</value>
          +    <description>Used in conjunction with hbase.ipc.server.callqueue.read.share
          +      will split the read call queues into small-read and long-read queues.
          +      A value of 0 or 1 indicate to use the same set of queues for gets and scans.
          +    </description>
          

          can we get to a place where scan or gets receive no share? What happens then?

          +    int numScanQueues = (int)Math.floor(numReadQueues * scanShare);
          +    int scanHandlers = (int)Math.floor(readHandlers * scanShare);
          +    if ((numReadQueues - numScanQueues) > 0) {
          +      numReadQueues -= numScanQueues;
          +      readHandlers -= scanHandlers;
          +    } else {
          +      numScanQueues = 0;
          +      scanHandlers = 0;
          +    }
          +
               this.writeHandlersCount = Math.max(writeHandlers, numWriteQueues);
               this.readHandlersCount = Math.max(readHandlers, numReadQueues);
          +    this.scanHandlersCount = Math.max(scanHandlers, numScanQueues);
               this.numWriteQueues = numWriteQueues;
               this.numReadQueues = numReadQueues;
          +    this.numScanQueues = numScanQueues;
          

          Need some validation to make sure negative values or other silly values throw warnings and don't cause problems.

              String callQueueType = conf.get(CALL_QUEUE_TYPE_CONF_KEY, CALL_QUEUE_TYPE_DEADLINE_CONF_VALUE);
               float callqReadShare = conf.getFloat(CALL_QUEUE_READ_SHARE_CONF_KEY, 0);
          +    float callqScanShare = conf.getFloat(CALL_QUEUE_SCAN_SHARE_CONF_KEY, 0);
          
          Show
          jmhsieh Jonathan Hsieh added a comment - Without reading the code, this is ambiguous - what does 0 or 1 specifically mean? what is the default setting? does 0 mean get+scan shared, and 1 mean separate? is it a float? (it is) I it a sub share of h.i.s.callqueue.read.share or is it at the "same level"? (it is not). (related question - is hbase.ipc.server.callqueue.read.share a float or is it specifically 0, 0.5 and 1.0 values?) + <name>hbase.ipc.server.callqueue.scan.share</name> + <value>0</value> + <description>Used in conjunction with hbase.ipc.server.callqueue.read.share + will split the read call queues into small-read and long -read queues. + A value of 0 or 1 indicate to use the same set of queues for gets and scans. + </description> can we get to a place where scan or gets receive no share? What happens then? + int numScanQueues = ( int ) Math .floor(numReadQueues * scanShare); + int scanHandlers = ( int ) Math .floor(readHandlers * scanShare); + if ((numReadQueues - numScanQueues) > 0) { + numReadQueues -= numScanQueues; + readHandlers -= scanHandlers; + } else { + numScanQueues = 0; + scanHandlers = 0; + } + this .writeHandlersCount = Math .max(writeHandlers, numWriteQueues); this .readHandlersCount = Math .max(readHandlers, numReadQueues); + this .scanHandlersCount = Math .max(scanHandlers, numScanQueues); this .numWriteQueues = numWriteQueues; this .numReadQueues = numReadQueues; + this .numScanQueues = numScanQueues; Need some validation to make sure negative values or other silly values throw warnings and don't cause problems. String callQueueType = conf.get(CALL_QUEUE_TYPE_CONF_KEY, CALL_QUEUE_TYPE_DEADLINE_CONF_VALUE); float callqReadShare = conf.getFloat(CALL_QUEUE_READ_SHARE_CONF_KEY, 0); + float callqScanShare = conf.getFloat(CALL_QUEUE_SCAN_SHARE_CONF_KEY, 0);
          Hide
          mbertozzi Matteo Bertozzi added a comment -

          Without reading the code, this is ambiguous - what does 0 or 1 specifically mean? what is the default setting? does 0 mean get+scan shared, and 1 mean separate? is it a float? (it is) I it a sub share of h.i.s.callqueue.read.share or is it at the "same level"? (it is not).

          Isn't that all explained in the hbase-default doc?
          A value of 0 or 1 indicate to use the same set of queues for gets and scans.
          any other value is multiplied by the number of read queues that you have.

          (related question - is hbase.ipc.server.callqueue.read.share a float or is it specifically 0, 0.5 and 1.0 values?)

          a float...

          can we get to a place where scan or gets receive no share? What happens then?

          no, you get at least 1 get queue in case you want lots of more scans. otherwise the queues are shared.
          Basically it does what you expect, "give me more scans" or "give me less scans" if you want to do the math to specify the right number of queues good for you, otherwise the will do what is right based on what you specified.

          Need some validation to make sure negative values or other silly values throw warnings and don't cause problems.

          aside the fact that I don't see validation in any place where we have conf values, here all the value that you set will result in a valid settings. < 0 will result in 0, > 0 will result in 1

          Show
          mbertozzi Matteo Bertozzi added a comment - Without reading the code, this is ambiguous - what does 0 or 1 specifically mean? what is the default setting? does 0 mean get+scan shared, and 1 mean separate? is it a float? (it is) I it a sub share of h.i.s.callqueue.read.share or is it at the "same level"? (it is not). Isn't that all explained in the hbase-default doc? A value of 0 or 1 indicate to use the same set of queues for gets and scans. any other value is multiplied by the number of read queues that you have. (related question - is hbase.ipc.server.callqueue.read.share a float or is it specifically 0, 0.5 and 1.0 values?) a float... can we get to a place where scan or gets receive no share? What happens then? no, you get at least 1 get queue in case you want lots of more scans. otherwise the queues are shared. Basically it does what you expect, "give me more scans" or "give me less scans" if you want to do the math to specify the right number of queues good for you, otherwise the will do what is right based on what you specified. Need some validation to make sure negative values or other silly values throw warnings and don't cause problems. aside the fact that I don't see validation in any place where we have conf values, here all the value that you set will result in a valid settings. < 0 will result in 0, > 0 will result in 1
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          Isn't that all explained in the hbase-default doc?
          A value of 0 or 1 indicate to use the same set of queues for gets and scans.
          any other value is multiplied by the number of read queues that you have.

          I'm asking for a better explanation in the hbase-default.xml description – if I were a user looking only at this file for how to set the value, I don't know if what valid values are. As worded "A value of 0 or 1 indicate to use the same set of queues for gets and scans." could be interpreted to mean that when I set the value to '0' queues are shared, and when I set the value to '1' the queues are also shared. It seems like could set it to 42 to turn off the get/scan sharing. The fact that it is a float ratio between 0 and 1 is important to note here.

          Basically it does what you expect, "give me more scans" or "give me less scans" if you want to do the math to specify the right number of queues good for you, otherwise the will do what is right based on what you specified.

          if I just look at the hbase-default.xml I don't know what to expect. I want to you make it clear for the user in the in hbase-default.xml

          aside the fact that I don't see validation in any place where we have conf values, here all the value that you set will result in a valid settings. < 0 will result in 0, > 0 will result in 1

          where is this enforced? I didn't see it in this patch and theese values get used in the calculations of # of counts. As a user I could put values in like -1 or 42 and not know it was silly.

          Show
          jmhsieh Jonathan Hsieh added a comment - Isn't that all explained in the hbase-default doc? A value of 0 or 1 indicate to use the same set of queues for gets and scans. any other value is multiplied by the number of read queues that you have. I'm asking for a better explanation in the hbase-default.xml description – if I were a user looking only at this file for how to set the value, I don't know if what valid values are. As worded "A value of 0 or 1 indicate to use the same set of queues for gets and scans." could be interpreted to mean that when I set the value to '0' queues are shared, and when I set the value to '1' the queues are also shared. It seems like could set it to 42 to turn off the get/scan sharing. The fact that it is a float ratio between 0 and 1 is important to note here. Basically it does what you expect, "give me more scans" or "give me less scans" if you want to do the math to specify the right number of queues good for you, otherwise the will do what is right based on what you specified. if I just look at the hbase-default.xml I don't know what to expect. I want to you make it clear for the user in the in hbase-default.xml aside the fact that I don't see validation in any place where we have conf values, here all the value that you set will result in a valid settings. < 0 will result in 0, > 0 will result in 1 where is this enforced? I didn't see it in this patch and theese values get used in the calculations of # of counts. As a user I could put values in like -1 or 42 and not know it was silly.
          Hide
          mbertozzi Matteo Bertozzi added a comment -

          As worded "A value of 0 or 1 indicate to use the same set of queues for gets and scans." could be interpreted to mean that when I set the value to '0' queues are shared, and when I set the value to '1' the queues are also shared.

          The interpretation is correct, both 0 and 1 means that you share the same queues..

          It seems like could set it to 42 to turn off the get/scan sharing. The fact that it is a float ratio between 0 and 1 is important to note here.

          You can do what ever you want.. but the fact that it says "share" it should point you to the fact that is a float.

          if I just look at the hbase-default.xml I don't know what to expect. I want to you make it clear for the user in the in hbase-default.xml

          Any help with the wording? to me "Used in conjunction with hbase.ipc.server.callqueue.read.share will split the read call queues into small-read and long-read queues" is already clear enough and after that there is also explained the 0 - 1 corner case... so if you have something to add feel free to post it.

          I didn't see it in this patch and theese values get used in the calculations of # of counts. As a user I could put values in like -1 or 42 and not know it was silly.

          Yeah, you can put 42 and it will be solved by "if ((numReadQueues - numScanQueues) > 0)"

          Show
          mbertozzi Matteo Bertozzi added a comment - As worded "A value of 0 or 1 indicate to use the same set of queues for gets and scans." could be interpreted to mean that when I set the value to '0' queues are shared, and when I set the value to '1' the queues are also shared. The interpretation is correct, both 0 and 1 means that you share the same queues.. It seems like could set it to 42 to turn off the get/scan sharing. The fact that it is a float ratio between 0 and 1 is important to note here. You can do what ever you want.. but the fact that it says "share" it should point you to the fact that is a float. if I just look at the hbase-default.xml I don't know what to expect. I want to you make it clear for the user in the in hbase-default.xml Any help with the wording? to me "Used in conjunction with hbase.ipc.server.callqueue.read.share will split the read call queues into small-read and long-read queues" is already clear enough and after that there is also explained the 0 - 1 corner case... so if you have something to add feel free to post it. I didn't see it in this patch and theese values get used in the calculations of # of counts. As a user I could put values in like -1 or 42 and not know it was silly. Yeah, you can put 42 and it will be solved by "if ((numReadQueues - numScanQueues) > 0)"
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          You can do what ever you want.. but the fact that it says "share" it should point you to the fact that is a float.

          Initially I interpreted share as a verb and thought it was boolean with 0 and 1.

          Any help with the wording? to me "Used in conjunction with hbase.ipc.server.callqueue.read.share will split the read call queues into small-read and long-read queues" is already clear enough and after that there is also explained the 0 - 1 corner case... so if you have something to add feel free to post it.

          Give examples in the hbase-defaults.xml for these settings – it would clarify some of the confusion I've encountered. Give an example settings for scenarios where you have gets and scans as equals, where gets get a lot more priority and where scans get a lot more priority. Probably want the same with writes vs gets/scans as well.

          Yeah, you can put 42 and it will be solved by "if ((numReadQueues - numScanQueues) > 0)"

          my point is that as a user you are asking me to look at the code. Let's assume I'm a clever user and looked at the code — you haven't made any of these invariants known. comments in the hbase-defaults or explicit checks in code would be helpful.

          Show
          jmhsieh Jonathan Hsieh added a comment - You can do what ever you want.. but the fact that it says "share" it should point you to the fact that is a float. Initially I interpreted share as a verb and thought it was boolean with 0 and 1. Any help with the wording? to me "Used in conjunction with hbase.ipc.server.callqueue.read.share will split the read call queues into small-read and long-read queues" is already clear enough and after that there is also explained the 0 - 1 corner case... so if you have something to add feel free to post it. Give examples in the hbase-defaults.xml for these settings – it would clarify some of the confusion I've encountered. Give an example settings for scenarios where you have gets and scans as equals, where gets get a lot more priority and where scans get a lot more priority. Probably want the same with writes vs gets/scans as well. Yeah, you can put 42 and it will be solved by "if ((numReadQueues - numScanQueues) > 0)" my point is that as a user you are asking me to look at the code. Let's assume I'm a clever user and looked at the code — you haven't made any of these invariants known. comments in the hbase-defaults or explicit checks in code would be helpful.
          Hide
          mbertozzi Matteo Bertozzi added a comment -

          Give examples in the hbase-defaults.xml for these settings – it would clarify some of the confusion I've encountered. Give an example settings for scenarios where you have gets and scans as equals, where gets get a lot more priority and where scans get a lot more priority. Probably want the same with writes vs gets/scans as well.

          The example of the 0.5 value in the read.share seems to have thrown you off...

          <name>hbase.ipc.server.callqueue.read.share</name>
          <description>Split the call queues into read and write queues.
                A value of 0 indicate to not split the call queues.
                A value of 0.5 means there will be the same number of read and write queues
                A value of 1.0 means that all the queues except one are used to dispatch read requests.
          </description>
          

          adding something like this? "A value > 0.5 means that there will be more read queues than write queues" and "A value < 0.5 means that there will be less read queues than write queues".

          <name>hbase.ipc.server.callqueue.scan.share</name>
          <value>0</value>
          <description>Used in conjunction with hbase.ipc.server.callqueue.read.share
            will split the read call queues into small-read and long-read queues.
            A value of 0 or 1 indicate to use the same set of queues for gets and scans.
          </description>
          

          Adding something like this? "A value of 0.5 means that there will be the same number of short-read and long-read queues" and "A value > 0.5 means that there will be more long-read queues then short-read queues" and "A value < 0.5 means that there will be less long-read queues then short-read queues"

          my point is that as a user you are asking me to look at the code. Let's assume I'm a clever user and looked at the code — you haven't made any of these invariants known. comments in the hbase-defaults or explicit checks in code would be helpful.

          Why? "share" may be interpreted as a boolean but if you see 0 as default instead of false, it is probably not a boolean.. then you read about the 0 and 1 edge case... and you may start understanding that that probably is the limit...
          otherwise, you are just the fancy user that wants to break things and set random crap as value...
          If you want I can add "the value must be between 0.0 and 1.0" to the doc... but you may want to add that also to all the other fields that we have in hbase-default.xml

          Show
          mbertozzi Matteo Bertozzi added a comment - Give examples in the hbase-defaults.xml for these settings – it would clarify some of the confusion I've encountered. Give an example settings for scenarios where you have gets and scans as equals, where gets get a lot more priority and where scans get a lot more priority. Probably want the same with writes vs gets/scans as well. The example of the 0.5 value in the read.share seems to have thrown you off... <name>hbase.ipc.server.callqueue.read.share</name> <description>Split the call queues into read and write queues. A value of 0 indicate to not split the call queues. A value of 0.5 means there will be the same number of read and write queues A value of 1.0 means that all the queues except one are used to dispatch read requests. </description> adding something like this? "A value > 0.5 means that there will be more read queues than write queues" and "A value < 0.5 means that there will be less read queues than write queues". <name>hbase.ipc.server.callqueue.scan.share</name> <value>0</value> <description>Used in conjunction with hbase.ipc.server.callqueue.read.share will split the read call queues into small-read and long -read queues. A value of 0 or 1 indicate to use the same set of queues for gets and scans. </description> Adding something like this? "A value of 0.5 means that there will be the same number of short-read and long-read queues" and "A value > 0.5 means that there will be more long-read queues then short-read queues" and "A value < 0.5 means that there will be less long-read queues then short-read queues" my point is that as a user you are asking me to look at the code. Let's assume I'm a clever user and looked at the code — you haven't made any of these invariants known. comments in the hbase-defaults or explicit checks in code would be helpful. Why? "share" may be interpreted as a boolean but if you see 0 as default instead of false, it is probably not a boolean.. then you read about the 0 and 1 edge case... and you may start understanding that that probably is the limit... otherwise, you are just the fancy user that wants to break things and set random crap as value... If you want I can add "the value must be between 0.0 and 1.0" to the doc... but you may want to add that also to all the other fields that we have in hbase-default.xml
          Hide
          jmhsieh Jonathan Hsieh added a comment - - edited

          Adding something like this? "A value of 0.5 means that there will be the same number of short-read and long-read queues" and "A value > 0.5 means that there will be more long-read queues then short-read queues" and "A value < 0.5 means that there will be less long-read queues then short-read queues"

          Yeah. Let me give some examples that could illustrate my confusion and help clarify.

          hbase.ipc.server.callqueue.read.share = 1.0
          hbase.ipc.server.callqueue.scan.share = .5

          Does this mean that writes are 0%, gets are 50% and scans are 50%?
          or Does this mean that writes are 0%, gets are 66% (1.0/(1.0+.5)) and scans are 33% (.5/(1.0+.5)?

          hbase.ipc.server.callqueue.read.share = .5
          hbase.ipc.server.callqueue.scan.share = .5

          Does this mean that writes are 0%, gets are 50% and scans are 50%?
          Or, does this mean that writes are 50%, gets are 25% and scans are 25%?

          Why? "share" may be interpreted as a boolean but if you see 0 as default instead of false, it is probably not a boolean.. then you read about the 0 and 1 edge case... and you may start understanding that that probably is the limit...
          otherwise, you are just the fancy user that wants to break things and set random crap as value...

          I'm not worried about fancy users, I'm worried about uninformed users that unknowingly break things because they don't know or don't get an indication that they set a crap value.

          If you want I can add "the value must be between 0.0 and 1.0" to the doc... but you may want to add that also to all the other fields that we have in hbase-default.xml

          Yes, I fact I strongly encourage this for other settings as well. It is even better if the code warns when there is a violation. If we can add more guardrails so uninformed users can't make silly mistakes (and get notified about it), hbase becomes easier to use and we avoid bad situations preemptively.

          Show
          jmhsieh Jonathan Hsieh added a comment - - edited Adding something like this? "A value of 0.5 means that there will be the same number of short-read and long-read queues" and "A value > 0.5 means that there will be more long-read queues then short-read queues" and "A value < 0.5 means that there will be less long-read queues then short-read queues" Yeah. Let me give some examples that could illustrate my confusion and help clarify. hbase.ipc.server.callqueue.read.share = 1.0 hbase.ipc.server.callqueue.scan.share = .5 Does this mean that writes are 0%, gets are 50% and scans are 50%? or Does this mean that writes are 0%, gets are 66% (1.0/(1.0+.5)) and scans are 33% (.5/(1.0+.5)? hbase.ipc.server.callqueue.read.share = .5 hbase.ipc.server.callqueue.scan.share = .5 Does this mean that writes are 0%, gets are 50% and scans are 50%? Or, does this mean that writes are 50%, gets are 25% and scans are 25%? Why? "share" may be interpreted as a boolean but if you see 0 as default instead of false, it is probably not a boolean.. then you read about the 0 and 1 edge case... and you may start understanding that that probably is the limit... otherwise, you are just the fancy user that wants to break things and set random crap as value... I'm not worried about fancy users, I'm worried about uninformed users that unknowingly break things because they don't know or don't get an indication that they set a crap value. If you want I can add "the value must be between 0.0 and 1.0" to the doc... but you may want to add that also to all the other fields that we have in hbase-default.xml Yes, I fact I strongly encourage this for other settings as well. It is even better if the code warns when there is a violation. If we can add more guardrails so uninformed users can't make silly mistakes (and get notified about it), hbase becomes easier to use and we avoid bad situations preemptively.
          Hide
          mbertozzi Matteo Bertozzi added a comment -

          read.share = 1.0 scan.share = .5 Does this mean that writes are 0%, gets are 50% and scans are 50%

          almost, but since 0% writes does not make sense you have at least 1 write queue...

          read.share = .5 .scan.share = .5
          Does this mean that writes are 0%, gets are 50% and scans are 50%?
          Or, does this mean that writes are 50%, gets are 25% and scans are 25%?

          The second one.. isn't this clear enough? "will split the READ CALL QUEUES into small-read and long-read queues".
          You split reads and writes, and then you split the scans. otherwise will result too complicate to understand

          can you wordify what you want to see in the doc? I'm a bit stuck here

          Show
          mbertozzi Matteo Bertozzi added a comment - read.share = 1.0 scan.share = .5 Does this mean that writes are 0%, gets are 50% and scans are 50% almost, but since 0% writes does not make sense you have at least 1 write queue... read.share = .5 .scan.share = .5 Does this mean that writes are 0%, gets are 50% and scans are 50%? Or, does this mean that writes are 50%, gets are 25% and scans are 25%? The second one.. isn't this clear enough? "will split the READ CALL QUEUES into small-read and long-read queues". You split reads and writes, and then you split the scans. otherwise will result too complicate to understand can you wordify what you want to see in the doc? I'm a bit stuck here
          Hide
          mbertozzi Matteo Bertozzi added a comment -

          Talked with jon offline, an example with numbers is probably easier to understand. Also renaming from .share to .ratio should make clearer that the value is an interval between 0 and 1

          Show
          mbertozzi Matteo Bertozzi added a comment - Talked with jon offline, an example with numbers is probably easier to understand. Also renaming from .share to .ratio should make clearer that the value is an interval between 0 and 1
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12661696/HBASE-11724-v1.patch
          against trunk revision .
          ATTACHMENT ID: 12661696

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels
          org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay

          -1 core zombie tests. There are 2 zombie test(s):

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12661696/HBASE-11724-v1.patch against trunk revision . ATTACHMENT ID: 12661696 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay -1 core zombie tests . There are 2 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10432//console This message is automatically generated.
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          Thanks, the example helps immensely!

          nit - fix this doc bug before commit.

          +      a scan.ratio of 0.8 means that: 6 queues will contain only long-read requests
          +      and 6 queues will contain only short-read requests.
          

          +1

          Show
          jmhsieh Jonathan Hsieh added a comment - Thanks, the example helps immensely! nit - fix this doc bug before commit. + a scan.ratio of 0.8 means that: 6 queues will contain only long -read requests + and 6 queues will contain only short -read requests. +1
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          +1 v2.

          Show
          jmhsieh Jonathan Hsieh added a comment - +1 v2.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12661715/HBASE-11724-v2.patch
          against trunk revision .
          ATTACHMENT ID: 12661715

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels
          org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12661715/HBASE-11724-v2.patch against trunk revision . ATTACHMENT ID: 12661715 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabels org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10433//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-1.0 #103 (See https://builds.apache.org/job/HBase-1.0/103/)
          HBASE-11724 Add to RWQueueRpcExecutor the ability to split get and scan handlers (matteo.bertozzi: rev 7df6f580010e1b934ba0cc2b14395185eff8948e)

          • hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java
          • hbase-common/src/main/resources/hbase-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-1.0 #103 (See https://builds.apache.org/job/HBase-1.0/103/ ) HBASE-11724 Add to RWQueueRpcExecutor the ability to split get and scan handlers (matteo.bertozzi: rev 7df6f580010e1b934ba0cc2b14395185eff8948e) hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java hbase-common/src/main/resources/hbase-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5400 (See https://builds.apache.org/job/HBase-TRUNK/5400/)
          HBASE-11724 Add to RWQueueRpcExecutor the ability to split get and scan handlers (matteo.bertozzi: rev e1e70a7e2f74c45e6b58b0ccace21969a7a29358)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java
          • hbase-common/src/main/resources/hbase-default.xml
          • hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5400 (See https://builds.apache.org/job/HBase-TRUNK/5400/ ) HBASE-11724 Add to RWQueueRpcExecutor the ability to split get and scan handlers (matteo.bertozzi: rev e1e70a7e2f74c45e6b58b0ccace21969a7a29358) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java hbase-common/src/main/resources/hbase-default.xml hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5414 (See https://builds.apache.org/job/HBase-TRUNK/5414/)
          HBASE-11737 Document callQueue improvements from HBASE-11355 and HBASE-11724 (Misty Stanley-Jones) (matteo.bertozzi: rev a55a65017cc182e3efd4639e3959af09f178d7d1)

          • src/main/docbkx/performance.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5414 (See https://builds.apache.org/job/HBase-TRUNK/5414/ ) HBASE-11737 Document callQueue improvements from HBASE-11355 and HBASE-11724 (Misty Stanley-Jones) (matteo.bertozzi: rev a55a65017cc182e3efd4639e3959af09f178d7d1) src/main/docbkx/performance.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-1.0 #115 (See https://builds.apache.org/job/HBase-1.0/115/)
          HBASE-11737 Document callQueue improvements from HBASE-11355 and HBASE-11724 (Misty Stanley-Jones) (matteo.bertozzi: rev 5c1ae840f21f7a3857543e408ef20a63be2b0751)

          • src/main/docbkx/performance.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-1.0 #115 (See https://builds.apache.org/job/HBase-1.0/115/ ) HBASE-11737 Document callQueue improvements from HBASE-11355 and HBASE-11724 (Misty Stanley-Jones) (matteo.bertozzi: rev 5c1ae840f21f7a3857543e408ef20a63be2b0751) src/main/docbkx/performance.xml
          Hide
          enis Enis Soztutar added a comment -

          Closing this issue after 0.99.0 release.

          Show
          enis Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

            People

            • Assignee:
              mbertozzi Matteo Bertozzi
              Reporter:
              mbertozzi Matteo Bertozzi
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development