[HDDS-11224] Increase hdds.datanode.handler.count

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDDS-7593
• Labels: pull-request-available
• Release Note: Increase hdds.datanode.handler.count default value from 1 to 10 to improve DataNode throughput performance.
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 23/Jul/24 21:52
• Updated: 02/Aug/24 22:00
• Resolved: 02/Aug/24 22:00

Description

hdds.datanode.handler.count is default to 1. Increasing to 10 improves DN Echo throughput

ozone freon dne --clients=32 -t 32 -n 1000000 --sleep-time-ms=0 --ratis --payload-req=0 --container-id=1 

before:

             count = 555484
         mean rate = 10309.69 calls/second
     1-minute rate = 9451.50 calls/second
     5-minute rate = 8537.91 calls/second
    15-minute rate = 8327.00 calls/second
               min = 1.29 milliseconds
               max = 34.97 milliseconds
              mean = 3.00 milliseconds
            stddev = 1.06 milliseconds
            median = 2.85 milliseconds
              75% <= 3.19 milliseconds
              95% <= 4.04 milliseconds
              98% <= 4.38 milliseconds
              99% <= 4.82 milliseconds
            99.9% <= 11.82 milliseconds

after setting hdds.datanode.handler.count to 10:

             count = 1000000
         mean rate = 20817.03 calls/second
     1-minute rate = 18431.88 calls/second
     5-minute rate = 16256.12 calls/second
    15-minute rate = 15770.61 calls/second
               min = 0.56 milliseconds
               max = 9.36 milliseconds
              mean = 1.50 milliseconds
            stddev = 0.60 milliseconds
            median = 1.38 milliseconds
              75% <= 1.69 milliseconds
              95% <= 2.47 milliseconds
              98% <= 2.82 milliseconds
              99% <= 3.23 milliseconds
            99.9% <= 7.58 milliseconds

Issue Links

• is a child of: HDDS-10755 [hsync] Investigate why DataNode Echo throughput is so low
• links to: GitHub Pull Request #7011