[HDDS-11225] Increase ipc.server.read.threadpool.size

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDDS-7593
• Labels: pull-request-available
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 23/Jul/24 22:03
• Updated: 12/Aug/24 06:37
• Resolved: 02/Aug/24 22:31

Description

OM Echo throughput is affected by this Hadoop configuration.

The default is 1. Increasing it to 10 improves throughput by about 4x.

default:

ozone freon ome --clients=10 -n 500000 -t 100

rpc-payload
             count = 500000
         mean rate = 35199.67 calls/second
     1-minute rate = 34028.60 calls/second
     5-minute rate = 33769.20 calls/second
    15-minute rate = 33724.26 calls/second
               min = 0.72 milliseconds
               max = 19.81 milliseconds
              mean = 2.61 milliseconds
            stddev = 1.32 milliseconds
            median = 2.50 milliseconds
              75% <= 2.86 milliseconds
              95% <= 3.54 milliseconds
              98% <= 4.28 milliseconds
              99% <= 9.44 milliseconds
            99.9% <= 19.81 milliseconds 

after:

ozone freon ome --clients=30 -n 5000000 -t 100

rpc-payload
             count = 5000000
         mean rate = 133982.86 calls/second
     1-minute rate = 111649.80 calls/second
     5-minute rate = 94971.80 calls/second
    15-minute rate = 91514.17 calls/second
               min = 0.18 milliseconds
               max = 13.46 milliseconds
              mean = 0.65 milliseconds
            stddev = 0.64 milliseconds
            median = 0.61 milliseconds
              75% <= 0.70 milliseconds
              95% <= 0.85 milliseconds
              98% <= 0.94 milliseconds
              99% <= 1.96 milliseconds
            99.9% <= 13.46 milliseconds 

Issue Links

• is a child of: HDDS-10755 [hsync] Investigate why DataNode Echo throughput is so low
• relates to: HDDS-11302 Ozone wrapper configurations to increase ipc.server.read.threadpool.size for SCM and Datanode
• links to: GitHub Pull Request #7007