[HDDS-7034] File handle leak on datanode in PutBlock

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.0
• Component/s: Ozone Datanode
• Labels: pull-request-available
• Target Version/s: 1.3.0
• Assignee: Attila Doroszlai
• Reporter: George Jahad
• Created: 22/Jul/22 06:33
• Updated: 23/Jul/22 07:07
• Resolved: 23/Jul/22 07:07

Description

This change seems to have caused a bad file handle leak on the datanodes:

 

https://github.com/apache/ozone/pull/3514/files#diff-04096cfd662f347dcf21e9e13e714af882dab2dfc02e4e9d71307b0c62b35a24

 

Reverting it, causes the problem to go away.  You can reproduce the problem like so:

 

 

cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/compose/ozone

docker-compose up --no-recreate --scale datanode=3 -d

 

Then watch the file handle count on a datanode:

 

docker exec -it ozone_datanode_1 bash
sudo bash
yum install -y lsof
while : ; do echo -n "$(date) open files: "; lsof -u hadoop | wc -l; sleep 3s; done

 

Meanwhile run freon from a different node:

docker exec -it ozone_om_1 bash

ozone freon ockg --volume=btest --bucket=freon --threads=24 --replication-type=RATIS --factor=THREE --size=1024 --number-of-tests=2000000 --prefix=small-file

 

and the datanode will look like this within a few minutes:

Thu Jul 21 22:42:45 UTC 2022 open files: 954
Thu Jul 21 22:42:48 UTC 2022 open files: 2184
Thu Jul 21 22:42:51 UTC 2022 open files: 3843
Thu Jul 21 22:42:54 UTC 2022 open files: 5838
Thu Jul 21 22:42:58 UTC 2022 open files: 8240
Thu Jul 21 22:43:01 UTC 2022 open files: 11827
Thu Jul 21 22:43:04 UTC 2022 open files: 16641
Thu Jul 21 22:43:08 UTC 2022 open files: 21931
Thu Jul 21 22:43:11 UTC 2022 open files: 27470
Thu Jul 21 22:43:15 UTC 2022 open files: 33239
Thu Jul 21 22:43:19 UTC 2022 open files: 39027
Thu Jul 21 22:43:23 UTC 2022 open files: 45251
Thu Jul 21 22:43:27 UTC 2022 open files: 51667
Thu Jul 21 22:43:31 UTC 2022 open files: 57262

 

 

 

Issue Links

• is caused by: HDDS-6794 EC: Analyze and add putBlock even on non writing node in the case of partial single stripe.
• links to: GitHub Pull Request #3616