[HDDS-9840] ozone freon dfsg takes very long time when HSYNC is enabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Component/s: Ozone CLI
• Assignee: Ashish Kumar
• Reporter: Pratyush Bhatt
• Created: 05/Dec/23 11:19
• Updated: 25/Jan/24 18:07
• Resolved: 25/Jan/24 18:07

Description

The command generates 1 100MB file in about 3 seconds when HSYNC is not passed.

ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n 1 --path ofs://ozone1701449533/vol2/bucket2/dfsg -s 102400000 -t 32  -p nonsync
23/12/05 11:11:29 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
23/12/05 11:11:29 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
23/12/05 11:11:29 INFO impl.MetricsSystemImpl: ozone-freon metrics system started
23/12/05 11:11:29 INFO freon.BaseFreonGenerator: Executing test with prefix nonsync and number-of-tests 1 0.00% |?                                                                                                    |  0/1 Time: 0:00:00|  Disabling FS cache: fs.ofs.impl.disable.cache
 0.00% |?                                                                                                    |  0/1 Time: 0:00:01|  23/12/05 11:11:30 INFO rpc.RpcClient: Creating Bucket: vol2/bucket2, with bucket layout FILE_SYSTEM_OPTIMIZED, om as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1
23/12/05 11:11:31 WARN impl.MetricsSystemImpl: ozone-freon metrics system already initialized!
23/12/05 11:11:31 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
 0.00% |?                                                                                                    |  0/1 Time: 0:00:02|
12/5/23 11:11:33 AM ============================================================-- Timers ----------------------------------------------------------------------
file-create
             count = 1
         mean rate = 0.46 calls/second
     1-minute rate = 0.00 calls/second
     5-minute rate = 0.00 calls/second
    15-minute rate = 0.00 calls/second
               min = 2018.26 milliseconds
               max = 2018.26 milliseconds
              mean = 2018.26 milliseconds
            stddev = 0.00 milliseconds
            median = 2018.26 milliseconds
              75% <= 2018.26 milliseconds
              95% <= 2018.26 milliseconds
              98% <= 2018.26 milliseconds
              99% <= 2018.26 milliseconds
            99.9% <= 2018.26 milliseconds
Total execution time (sec): 3
Failures: 0
Successful executions: 1 

But when HSYNC flag is passed, the same command takes 161 seconds even for a single 1MB file.

ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n 1 --path ofs://ozone1701449533/vol3/bucket3/dfsg -s 1024000 -t 32 --sync HSYNC -p sync2
23/12/05 11:17:50 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
23/12/05 11:17:50 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
23/12/05 11:17:50 INFO impl.MetricsSystemImpl: ozone-freon metrics system started
23/12/05 11:17:50 INFO freon.BaseFreonGenerator: Executing test with prefix sync2 and number-of-tests 1

 0.00% |?                                                                                                    |  0/1 Time: 0:00:00|  Disabling FS cache: fs.ofs.impl.disable.cache
 0.00% |?                                                                                                    |  0/1 Time: 0:00:01|  23/12/05 11:17:51 WARN impl.MetricsSystemImpl: ozone-freon metrics system already initialized!
23/12/05 11:17:51 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
 0.00% |?                                                                                                    |  0/1 Time: 0:00:02|
12/5/23 11:20:31 AM ============================================================

-- Timers ----------------------------------------------------------------------
file-create
             count = 1
         mean rate = 0.01 calls/second
     1-minute rate = 0.00 calls/second
     5-minute rate = 0.00 calls/second
    15-minute rate = 0.00 calls/second
               min = 160038.05 milliseconds
               max = 160038.05 milliseconds
              mean = 160038.05 milliseconds
            stddev = 0.00 milliseconds
            median = 160038.05 milliseconds
              75% <= 160038.05 milliseconds
              95% <= 160038.05 milliseconds
              98% <= 160038.05 milliseconds
              99% <= 160038.05 milliseconds
            99.9% <= 160038.05 milliseconds


Total execution time (sec): 161
Failures: 0
Successful executions: 1ozone freon dfsg --buffer=1024 --copy-buffer=1024 -n 1 --path ofs://ozone1701449533/vol3/bucket3/dfsg -s 1024000 -t 32 --sync HSYNC -p sync2
23/12/05 11:17:50 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-ozone-freon.properties,hadoop-metrics2.properties
23/12/05 11:17:50 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
23/12/05 11:17:50 INFO impl.MetricsSystemImpl: ozone-freon metrics system started
23/12/05 11:17:50 INFO freon.BaseFreonGenerator: Executing test with prefix sync2 and number-of-tests 1 0.00% |?                                                                                                    |  0/1 Time: 0:00:00|  Disabling FS cache: fs.ofs.impl.disable.cache
 0.00% |?                                                                                                    |  0/1 Time: 0:00:01|  23/12/05 11:17:51 WARN impl.MetricsSystemImpl: ozone-freon metrics system already initialized!
23/12/05 11:17:51 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
 0.00% |?                                                                                                    |  0/1 Time: 0:00:02|
12/5/23 11:20:31 AM ============================================================-- Timers ----------------------------------------------------------------------
file-create
             count = 1
         mean rate = 0.01 calls/second
     1-minute rate = 0.00 calls/second
     5-minute rate = 0.00 calls/second
    15-minute rate = 0.00 calls/second
               min = 160038.05 milliseconds
               max = 160038.05 milliseconds
              mean = 160038.05 milliseconds
            stddev = 0.00 milliseconds
            median = 160038.05 milliseconds
              75% <= 160038.05 milliseconds
              95% <= 160038.05 milliseconds
              98% <= 160038.05 milliseconds
              99% <= 160038.05 milliseconds
            99.9% <= 160038.05 milliseconds
Total execution time (sec): 161
Failures: 0
Successful executions: 1

Issue Links

• is superceded by: HDDS-9752 [hsync] Make Putblock performance acceptable - Client side