Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.0
    • Component/s: Performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This issue is a continuation of HBASE-1938 work (That issue is closed). This issue is about getting the last patch posted by nkeywal over in HBASE-1938 applied (assigned nkeywal since he's done the work).

        Activity

        Hide
        stack added a comment -

        Here is the patch we are talking about: https://issues.apache.org/jira/secure/attachment/12489104/20110802_MemStore.patch

        Patch looks good. Running 50 TestHRegions on 0.90 in a row....

        Show
        stack added a comment - Here is the patch we are talking about: https://issues.apache.org/jira/secure/attachment/12489104/20110802_MemStore.patch Patch looks good. Running 50 TestHRegions on 0.90 in a row....
        Hide
        stack added a comment -

        So, it looks like timing difference seen by Andrew is no longer present: with patch loads happen faster and though the timings have wide variance, after a few runs, with patch we seem to faster.

        Here is a run of MemStoreScanPerformance before I applied above patch:

        stack-7:0.90 Stack$ ./bin/hbase org.apache.hadoop.hbase.regionserver.MemStoreScanPerformance
        JUnit version 4.8.1
        .Loaded in 2479 ms
        Scan with size 50000: 177 ms
        Scan with size 50000: 185 ms
        Scan with size 50000: 234 ms
        Scan with size 50000: 86 ms
        Scan with size 50000: 115 ms
        Scan with size 50000: 92 ms
        Scan with size 50000: 53 ms
        Scan with size 50000: 58 ms
        Scan with size 50000: 69 ms
        Scan with size 50000: 61 ms
        Loaded in 3134 ms
        Scan with size 75000: 77 ms
        Scan with size 75000: 87 ms
        Scan with size 75000: 88 ms
        Scan with size 75000: 84 ms
        Scan with size 75000: 71 ms
        Scan with size 75000: 110 ms
        Scan with size 75000: 109 ms
        Scan with size 75000: 100 ms
        Scan with size 75000: 114 ms
        Scan with size 75000: 111 ms
        Loaded in 4748 ms
        Scan with size 100000: 101 ms
        Scan with size 100000: 85 ms
        Scan with size 100000: 75 ms
        Scan with size 100000: 91 ms
        Scan with size 100000: 106 ms
        Scan with size 100000: 84 ms
        Scan with size 100000: 100 ms
        Scan with size 100000: 91 ms
        Scan with size 100000: 102 ms
        Scan with size 100000: 81 ms
        Loaded in 5904 ms
        Scan with size 125000: 138 ms
        Scan with size 125000: 175 ms
        Scan with size 125000: 163 ms
        Scan with size 125000: 145 ms
        Scan with size 125000: 150 ms
        Scan with size 125000: 122 ms
        Scan with size 125000: 112 ms
        Scan with size 125000: 118 ms
        Scan with size 125000: 146 ms
        Scan with size 125000: 136 ms
        Loaded in 6125 ms
        Scan with size 150000: 126 ms
        Scan with size 150000: 106 ms
        Scan with size 150000: 117 ms
        Scan with size 150000: 119 ms
        Scan with size 150000: 118 ms
        Scan with size 150000: 118 ms
        Scan with size 150000: 126 ms
        Scan with size 150000: 122 ms
        Scan with size 150000: 127 ms
        Scan with size 150000: 117 ms
        Total load time: 22390 ms (i.e:22 seconds)
        Total scan time: 5597 ms (i.e:5 seconds)
        Rows scanned per seconds: 893335
        Rows loaded per seconds: 223313
        
        Time: 28.787
        
        OK (1 test)
        

        Here is afterward:

        stack-7:0.90 Stack$ ./bin/hbase org.apache.hadoop.hbase.regionserver.MemStoreScanPerformance
        JUnit version 4.8.1
        .Loaded in 2293 ms
        Scan with size 50000: 147 ms
        Scan with size 50000: 195 ms
        Scan with size 50000: 164 ms
        Scan with size 50000: 88 ms
        Scan with size 50000: 98 ms
        Scan with size 50000: 80 ms
        Scan with size 50000: 86 ms
        Scan with size 50000: 57 ms
        Scan with size 50000: 61 ms
        Scan with size 50000: 48 ms
        Loaded in 2447 ms
        Scan with size 75000: 54 ms
        Scan with size 75000: 94 ms
        Scan with size 75000: 60 ms
        Scan with size 75000: 61 ms
        Scan with size 75000: 75 ms
        Scan with size 75000: 59 ms
        Scan with size 75000: 69 ms
        Scan with size 75000: 71 ms
        Scan with size 75000: 83 ms
        Scan with size 75000: 57 ms
        Loaded in 3381 ms
        Scan with size 100000: 81 ms
        Scan with size 100000: 84 ms
        Scan with size 100000: 84 ms
        Scan with size 100000: 85 ms
        Scan with size 100000: 83 ms
        Scan with size 100000: 85 ms
        Scan with size 100000: 108 ms
        Scan with size 100000: 78 ms
        Scan with size 100000: 85 ms
        Scan with size 100000: 75 ms
        Loaded in 4395 ms
        Scan with size 125000: 102 ms
        Scan with size 125000: 104 ms
        Scan with size 125000: 106 ms
        Scan with size 125000: 88 ms
        Scan with size 125000: 109 ms
        Scan with size 125000: 99 ms
        Scan with size 125000: 99 ms
        Scan with size 125000: 110 ms
        Scan with size 125000: 105 ms
        Scan with size 125000: 105 ms
        Loaded in 6552 ms
        Scan with size 150000: 110 ms
        Scan with size 150000: 112 ms
        Scan with size 150000: 107 ms
        Scan with size 150000: 126 ms
        Scan with size 150000: 114 ms
        Scan with size 150000: 118 ms
        Scan with size 150000: 131 ms
        Scan with size 150000: 110 ms
        Scan with size 150000: 113 ms
        Scan with size 150000: 117 ms
        Total load time: 19068 ms (i.e:19 seconds)
        Total scan time: 4731 ms (i.e:4 seconds)
        Rows scanned per seconds: 1056859
        Rows loaded per seconds: 262219
        
        Time: 24.52
        
        OK (1 test)
        
        stack-7:0.90 Stack$ 
        
        Show
        stack added a comment - So, it looks like timing difference seen by Andrew is no longer present: with patch loads happen faster and though the timings have wide variance, after a few runs, with patch we seem to faster. Here is a run of MemStoreScanPerformance before I applied above patch: stack-7:0.90 Stack$ ./bin/hbase org.apache.hadoop.hbase.regionserver.MemStoreScanPerformance JUnit version 4.8.1 .Loaded in 2479 ms Scan with size 50000: 177 ms Scan with size 50000: 185 ms Scan with size 50000: 234 ms Scan with size 50000: 86 ms Scan with size 50000: 115 ms Scan with size 50000: 92 ms Scan with size 50000: 53 ms Scan with size 50000: 58 ms Scan with size 50000: 69 ms Scan with size 50000: 61 ms Loaded in 3134 ms Scan with size 75000: 77 ms Scan with size 75000: 87 ms Scan with size 75000: 88 ms Scan with size 75000: 84 ms Scan with size 75000: 71 ms Scan with size 75000: 110 ms Scan with size 75000: 109 ms Scan with size 75000: 100 ms Scan with size 75000: 114 ms Scan with size 75000: 111 ms Loaded in 4748 ms Scan with size 100000: 101 ms Scan with size 100000: 85 ms Scan with size 100000: 75 ms Scan with size 100000: 91 ms Scan with size 100000: 106 ms Scan with size 100000: 84 ms Scan with size 100000: 100 ms Scan with size 100000: 91 ms Scan with size 100000: 102 ms Scan with size 100000: 81 ms Loaded in 5904 ms Scan with size 125000: 138 ms Scan with size 125000: 175 ms Scan with size 125000: 163 ms Scan with size 125000: 145 ms Scan with size 125000: 150 ms Scan with size 125000: 122 ms Scan with size 125000: 112 ms Scan with size 125000: 118 ms Scan with size 125000: 146 ms Scan with size 125000: 136 ms Loaded in 6125 ms Scan with size 150000: 126 ms Scan with size 150000: 106 ms Scan with size 150000: 117 ms Scan with size 150000: 119 ms Scan with size 150000: 118 ms Scan with size 150000: 118 ms Scan with size 150000: 126 ms Scan with size 150000: 122 ms Scan with size 150000: 127 ms Scan with size 150000: 117 ms Total load time: 22390 ms (i.e:22 seconds) Total scan time: 5597 ms (i.e:5 seconds) Rows scanned per seconds: 893335 Rows loaded per seconds: 223313 Time: 28.787 OK (1 test) Here is afterward: stack-7:0.90 Stack$ ./bin/hbase org.apache.hadoop.hbase.regionserver.MemStoreScanPerformance JUnit version 4.8.1 .Loaded in 2293 ms Scan with size 50000: 147 ms Scan with size 50000: 195 ms Scan with size 50000: 164 ms Scan with size 50000: 88 ms Scan with size 50000: 98 ms Scan with size 50000: 80 ms Scan with size 50000: 86 ms Scan with size 50000: 57 ms Scan with size 50000: 61 ms Scan with size 50000: 48 ms Loaded in 2447 ms Scan with size 75000: 54 ms Scan with size 75000: 94 ms Scan with size 75000: 60 ms Scan with size 75000: 61 ms Scan with size 75000: 75 ms Scan with size 75000: 59 ms Scan with size 75000: 69 ms Scan with size 75000: 71 ms Scan with size 75000: 83 ms Scan with size 75000: 57 ms Loaded in 3381 ms Scan with size 100000: 81 ms Scan with size 100000: 84 ms Scan with size 100000: 84 ms Scan with size 100000: 85 ms Scan with size 100000: 83 ms Scan with size 100000: 85 ms Scan with size 100000: 108 ms Scan with size 100000: 78 ms Scan with size 100000: 85 ms Scan with size 100000: 75 ms Loaded in 4395 ms Scan with size 125000: 102 ms Scan with size 125000: 104 ms Scan with size 125000: 106 ms Scan with size 125000: 88 ms Scan with size 125000: 109 ms Scan with size 125000: 99 ms Scan with size 125000: 99 ms Scan with size 125000: 110 ms Scan with size 125000: 105 ms Scan with size 125000: 105 ms Loaded in 6552 ms Scan with size 150000: 110 ms Scan with size 150000: 112 ms Scan with size 150000: 107 ms Scan with size 150000: 126 ms Scan with size 150000: 114 ms Scan with size 150000: 118 ms Scan with size 150000: 131 ms Scan with size 150000: 110 ms Scan with size 150000: 113 ms Scan with size 150000: 117 ms Total load time: 19068 ms (i.e:19 seconds) Total scan time: 4731 ms (i.e:4 seconds) Rows scanned per seconds: 1056859 Rows loaded per seconds: 262219 Time: 24.52 OK (1 test) stack-7:0.90 Stack$
        Hide
        stack added a comment -

        Here is MemStoreScanPerformance with apache license. We should add this when we commit this patch.

        Show
        stack added a comment - Here is MemStoreScanPerformance with apache license. We should add this when we commit this patch.
        Hide
        stack added a comment -

        Hmm... so I still see fails. Out of 42 runs, three failed which is less than previous....

        Show
        stack added a comment - Hmm... so I still see fails. Out of 42 runs, three failed which is less than previous....
        Hide
        stack added a comment -

        Did more. 8 of 60 runs failed. All were testWritesWhileGetting failures.

        Show
        stack added a comment - Did more. 8 of 60 runs failed. All were testWritesWhileGetting failures.
        Hide
        Nicolas Liochon added a comment -

        Yes, I would expect them to be the scenario with consistency between read
        and flush mentioned in the bug
        HBASE-4195<https://issues.apache.org/jira/browse/HBASE-4195>(and
        already managed in
        HBASE-2856 <https://issues.apache.org/jira/browse/HBASE-2856>). You can
        check this by changing "int flushInterval = 10;" in the method
        "testWritesWhileGetting", for example by setting it to 2 or 3. This should
        increase the number of failure. As well, the MemStoreTS of the KV in error
        will be 0 (it comes from a file). This error (or at least the error I
        analyzed) comes from the fact that the flush is finished before the end of
        the Store#next, and not directly to the MemStore.

        A xarning as well the patch in
        HBASE-4195<https://issues.apache.org/jira/browse/HBASE-4195>will
        conflict with this one, because they both touch the "seek" & "reseek"
        function (even if it is actually different parts that are modified). Nothing
        complex, but I won't be able to do the followup on this until end of next
        week.

        Show
        Nicolas Liochon added a comment - Yes, I would expect them to be the scenario with consistency between read and flush mentioned in the bug HBASE-4195 < https://issues.apache.org/jira/browse/HBASE-4195 >(and already managed in HBASE-2856 < https://issues.apache.org/jira/browse/HBASE-2856 >). You can check this by changing "int flushInterval = 10;" in the method "testWritesWhileGetting", for example by setting it to 2 or 3. This should increase the number of failure. As well, the MemStoreTS of the KV in error will be 0 (it comes from a file). This error (or at least the error I analyzed) comes from the fact that the flush is finished before the end of the Store#next, and not directly to the MemStore. A xarning as well the patch in HBASE-4195 < https://issues.apache.org/jira/browse/HBASE-4195 >will conflict with this one, because they both touch the "seek" & "reseek" function (even if it is actually different parts that are modified). Nothing complex, but I won't be able to do the followup on this until end of next week.
        Hide
        stack added a comment -

        I changed the int flushInterval to 3 and now I get no failures. Trying 5.

        Show
        stack added a comment - I changed the int flushInterval to 3 and now I get no failures. Trying 5.
        Hide
        stack added a comment -

        Same for 5. No errors. Putting it back to 10.

        Show
        stack added a comment - Same for 5. No errors. Putting it back to 10.
        Hide
        Nicolas Liochon added a comment -

        See HBASE-4195. It's the same test.

        Show
        Nicolas Liochon added a comment - See HBASE-4195 . It's the same test.
        Hide
        Nicolas Liochon added a comment -

        Patch for this jira bug is included in HBASE-4195.

        Show
        Nicolas Liochon added a comment - Patch for this jira bug is included in HBASE-4195 .
        Hide
        stack added a comment -

        Resolving as fixed by hbase-4195 application.

        Show
        stack added a comment - Resolving as fixed by hbase-4195 application.

          People

          • Assignee:
            Nicolas Liochon
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development