HBase
  1. HBase
  2. HBASE-10997

Add a modulo argument to PE to constrain the key range

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Helps w/ keeping PE inside block cache but same number of clients.

      1. revert.txt
        4 kB
        stack
      2. modulo.txt
        4 kB
        stack

        Activity

        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #5123 (See https://builds.apache.org/job/HBase-TRUNK/5123/)
        HBASE-10997 Add a modulo argument to PE to constrain the key range; REVERT (stack: rev 1590864)

        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5123 (See https://builds.apache.org/job/HBase-TRUNK/5123/ ) HBASE-10997 Add a modulo argument to PE to constrain the key range; REVERT (stack: rev 1590864) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
        stack made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        stack added a comment -

        Closing to shut it down.

        Show
        stack added a comment - Closing to shut it down.
        stack made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Fix Version/s 0.99.0 [ 12325675 ]
        Resolution Fixed [ 1 ]
        Hide
        stack added a comment -

        Reverted this patch. Removed fix version. Shouldn't show up in any release.

        Show
        stack added a comment - Reverted this patch. Removed fix version. Shouldn't show up in any release.
        stack made changes -
        Attachment revert.txt [ 12642401 ]
        Hide
        stack added a comment -

        My revert.

        Show
        stack added a comment - My revert.
        stack made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        stack added a comment -

        Going to revert this. The --size argument does what this adds better.

        Show
        stack added a comment - Going to revert this. The --size argument does what this adds better.
        Hide
        stack added a comment -

        I'm not sure how this is useful.

        I use it like this.

        I want to run a PE that does random reads and that runs for a good while, long enough for the latency pattern to emerge. I want to run ten clients. Block cache is small. I want the random reads to stay in block cache. As is, the random reads range over a namespace that is clients X rows. To stay inside block cache, I can set rows small but my test is over before a pattern is well established. If I set rows to 1B, then clients will get 1B rows but if I add the modulo, the keys will be confined to the lower part of the range (dependent on what modulo is).

        If I want to bust block cache but stay inside the os cache, I up the modulo till just before i/o comes on.

        I like your 'size' better than my modulo. I'll remove this if your size works. Thanks.

        Sorry for committing this w/o waiting on review. Didn't think anyone cared (missed your size issue).

        Show
        stack added a comment - I'm not sure how this is useful. I use it like this. I want to run a PE that does random reads and that runs for a good while, long enough for the latency pattern to emerge. I want to run ten clients. Block cache is small. I want the random reads to stay in block cache. As is, the random reads range over a namespace that is clients X rows. To stay inside block cache, I can set rows small but my test is over before a pattern is well established. If I set rows to 1B, then clients will get 1B rows but if I add the modulo, the keys will be confined to the lower part of the range (dependent on what modulo is). If I want to bust block cache but stay inside the os cache, I up the modulo till just before i/o comes on. I like your 'size' better than my modulo. I'll remove this if your size works. Thanks. Sorry for committing this w/o waiting on review. Didn't think anyone cared (missed your size issue).
        Hide
        Nick Dimiduk added a comment -

        HBASE-9953 lets me think in terms of total GiB data size instead of ((rows * clients) / million rows-per-GiB). Given a total target size and a known rowsize, it will fill in the blank for --rows.

        This one limits the range of values a client will use. Sorry, caffeine hasn't set in yet; I'm not sure how this is useful. I guess it's for hotspotting whichever regions are hosting those first $(modulo) rows, is that it?

        In short, I think they're compatible.

        Show
        Nick Dimiduk added a comment - HBASE-9953 lets me think in terms of total GiB data size instead of ((rows * clients) / million rows-per-GiB). Given a total target size and a known rowsize, it will fill in the blank for --rows. This one limits the range of values a client will use. Sorry, caffeine hasn't set in yet; I'm not sure how this is useful. I guess it's for hotspotting whichever regions are hosting those first $(modulo) rows, is that it? In short, I think they're compatible.
        Hide
        stack added a comment -

        Nick Dimiduk Don't know about supercede. I missed hbase-9953. The modulo is a bit more basic. All random read clients were picking at random from the total namespace. The module narrows the namespace that clients operate in. If module is 10, then all clients will pick rows in range 0-10 even though rows might be 1M and clients 1000.

        Does that supercede? The patches look to do similar things. Your size confines where we operate in a similar way? (I can back this out np. You can imagine why I need this).

        Show
        stack added a comment - Nick Dimiduk Don't know about supercede. I missed hbase-9953. The modulo is a bit more basic. All random read clients were picking at random from the total namespace. The module narrows the namespace that clients operate in. If module is 10, then all clients will pick rows in range 0-10 even though rows might be 1M and clients 1000. Does that supercede? The patches look to do similar things. Your size confines where we operate in a similar way? (I can back this out np. You can imagine why I need this).
        Hide
        Nick Dimiduk added a comment -

        Hey boss, does this supersede HBASE-9953?

        Show
        Nick Dimiduk added a comment - Hey boss, does this supersede HBASE-9953 ?
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #5089 (See https://builds.apache.org/job/HBase-TRUNK/5089/)
        HBASE-10997 Add a modulo argument to PE to constrain the key range (stack: rev 1587801)

        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #5089 (See https://builds.apache.org/job/HBase-TRUNK/5089/ ) HBASE-10997 Add a modulo argument to PE to constrain the key range (stack: rev 1587801) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
        stack made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.99.0 [ 12325675 ]
        Resolution Fixed [ 1 ]
        Hide
        stack added a comment -

        Committed to trunk. Helps w/ my testing.

        Show
        stack added a comment - Committed to trunk. Helps w/ my testing.
        stack made changes -
        Field Original Value New Value
        Attachment modulo.txt [ 12640387 ]
        Hide
        stack added a comment -

        Small patch

        Show
        stack added a comment - Small patch
        stack created issue -

          People

          • Assignee:
            stack
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development