Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
Description
Ozone key put fails during parallel write with space quota set
Files being written - 10 No 's
File Size - 768 MB
Quota set of volume & bucket : 12.08 GB
Volume
[root@quasar-tebynh-1 ~]# ozone sh volume info o3://ozone1/vol-quota-1684404995/ { "metadata" : { }, "name" : "vol-quota-1684404995", "admin" : "hrt_qa", "owner" : "root", "quotaInBytes" : 12079595520, "quotaInNamespace" : -1, "usedNamespace" : 1, "creationTime" : "2023-05-18T10:16:40.765Z", "modificationTime" : "2023-05-18T10:16:59.288Z", "acls" : [ { "type" : "USER", "name" : "root", "aclScope" : "ACCESS", "aclList" : [ "ALL" ] }, { "type" : "GROUP", "name" : "root", "aclScope" : "ACCESS", "aclList" : [ "ALL" ] } ], "refCount" : 0 }
Bucket
[root@quasar-tebynh-1 ~]# ozone sh bucket info o3://ozone1/vol-quota-1684404995/buck-quota-1684404995 { "metadata" : { }, "volumeName" : "vol-quota-1684404995", "name" : "buck-quota-1684404995", "storageType" : "DISK", "versioning" : false, "usedBytes" : 11513364480, "usedNamespace" : 5, "creationTime" : "2023-05-18T10:16:44.333Z", "modificationTime" : "2023-05-18T10:16:51.701Z", "quotaInBytes" : 12079595520, "quotaInNamespace" : -1, "bucketLayout" : "FILE_SYSTEM_OPTIMIZED", "link" : false }
Keys successfully written
[root@quasar-tebynh-1 ~]# ozone fs -ls -r ofs://ozone1/vol-quota-1684404995/buck-quota-1684404995/ Found 5 items -rw-rw-rw- 3 hdfs hdfs 767557632 2023-05-18 10:20 ofs://ozone1/vol-quota-1684404995/buck-quota-1684404995/quota9_1684405023 -rw-rw-rw- 3 hdfs hdfs 767557632 2023-05-18 10:20 ofs://ozone1/vol-quota-1684404995/buck-quota-1684404995/quota6_1684405023 -rw-rw-rw- 3 hdfs hdfs 767557632 2023-05-18 10:20 ofs://ozone1/vol-quota-1684404995/buck-quota-1684404995/quota5_1684405023 -rw-rw-rw- 3 hdfs hdfs 767557632 2023-05-18 10:20 ofs://ozone1/vol-quota-1684404995/buck-quota-1684404995/quota3_1684405023 -rw-rw-rw- 3 hdfs hdfs 767557632 2023-05-18 10:20 ofs://ozone1/vol-quota-1684404995/buck-quota-1684404995/quota2_1684405023
While the parallel writes are happening 5 keys are getting written and other 5 are failing with Quota exceeded error.
Error Message for failing put's:
AssertionError: Ozone put Key failed with output=[23/05/18 10:19:17 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
QUOTA_EXCEEDED The DiskSpace quota of bucket:buck-quota-1684404995 exceeded quotaInBytes: 12079595520 Bytes but diskspace consumed: 13816037376 Bytes.]
