Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15405

Synchronize final results logging single thread in PE, fix wrong defaults in help message

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 0.98.19, 2.0.0
    • Component/s: Performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Corrects wrong default values for few options in the help message.

      Final stats from multiple clients are intermingled making it hard to understand. Also the logged stats aren't very machine readable. It can be helpful in a daily perf testing rig which scraps logs for results.

      Example of logs before the change.

      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576, latency mean=953.98, min=359.00, max=324050.00, stdDev=851.82, 95th=1368.00, 99th=1625.00
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576, latency mean=953.92, min=356.00, max=323394.00, stdDev=817.55, 95th=1370.00, 99th=1618.00
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576, latency mean=953.98, min=367.00, max=322745.00, stdDev=840.43, 95th=1369.00, 99th=1622.00
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log (microseconds), on 1048576 measures
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min      = 375.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min      = 363.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg      = 953.6624126434326
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg      = 953.4124526977539
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev   = 781.3929776087633
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev   = 742.8027916717297
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th     = 894.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th     = 894.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th     = 1070.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th     = 1071.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th     = 1369.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th     = 1369.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th     = 1623.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th     = 1624.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min      = 372.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th   = 3013.9980000000214
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg      = 953.2451229095459
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th   = 3043.9980000000214
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev   = 725.4744472152282
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th  = 25282.380199996755
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th     = 895.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th  = 25812.763399994
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th     = 1071.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.999th = 89772.78990004538
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th     = 1369.0
      16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.999th = 122808.39587019826
      

      After the change

      16/03/06 02:38:21 INFO hbase.PerformanceEvaluation: Test : RandomWriteTest, Thread : TestClient-1
      16/03/06 02:38:21 INFO hbase.PerformanceEvaluation: Latency (us) : , mean=6.23, min=2.00, max=101433.00, stdDev=246.62, 50th=2.00, 75th=2.00, 95th=3.00, 99th=13.00, 99.9th=558.00, 99.99th=9656.19, 99.999th=20213.63
      16/03/06 02:38:21 INFO hbase.PerformanceEvaluation: Num measures (latency) : 1048576
      16/03/06 02:38:21 INFO hbase.PerformanceEvaluation: ValueSize (bytes) : , mean=0.00, min=0.00, max=0.00, stdDev=0.00, 50th=0.00, 75th=0.00, 95th=0.00, 99th=0.00, 99.9th=0.00, 99.99th=0.00, 99.999th=0.00
      16/03/06 02:38:21 INFO hbase.PerformanceEvaluation: Num measures (ValueSize): 0
      ...
      16/03/06 02:38:21 INFO hbase.PerformanceEvaluation: Test : RandomWriteTest, Thread : TestClient-0
      16/03/06 02:38:21 INFO hbase.PerformanceEvaluation: Latency (us) : , mean=6.23, min=1.00, max=99321.00, stdDev=246.41, 50th=2.00, 75th=2.00, 95th=3.00, 99th=13.00, 99.9th=566.00, 99.99th=9694.60, 99.999th=23311.86
      ....
      

        Attachments

        1. HBASE-15405-branch-1.patch
          14 kB
          Apekshit Sharma
        2. HBASE-15405-master.patch
          10 kB
          Apekshit Sharma
        3. HBASE-15405-master.patch
          10 kB
          Apekshit Sharma
        4. HBASE-15405-master-v2.patch
          12 kB
          Apekshit Sharma
        5. HBASE-15405-master-v3.patch
          12 kB
          Apekshit Sharma
        6. HBASE-15405-master-v4.patch
          12 kB
          Michael Stack
        7. HBASE-15405-master-v4.patch
          12 kB
          Apekshit Sharma
        8. HBASE-15405-master-v4 (1).patch
          12 kB
          Michael Stack
        9. HBASE-15405-master-v4 (1).patch
          12 kB
          Michael Stack
        10. HBASE-15405-master-v5.patch
          12 kB
          Apekshit Sharma
        11. HBASE-15405-master-v6.patch
          12 kB
          Apekshit Sharma

          Activity

            People

            • Assignee:
              appy Apekshit Sharma
              Reporter:
              appy Apekshit Sharma
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: