[HDDS-2331] Client OOME due to buffer retention

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Component/s: Ozone Client
• Target Version/s: 0.5.0
• Assignee: Shashikant Banerjee
• Reporter: Attila Doroszlai
• Created: 18/Oct/19 11:33
• Updated: 31/Oct/19 10:45
• Resolved: 31/Oct/19 10:45

Description

Freon random key generator exhausts default heap after just few hundred 1MB keys. Heap dump on OOME reveals 150+ instances of ContainerCommandRequestMessage, each with 16MB byte[].

Steps to reproduce:

1. Start Ozone cluster with 1 datanode
2. Start Freon (5K keys of size 1MB)

Result: OOME after a few hundred keys

$ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
$ docker-compose up -d
$ docker-compose exec scm bash
$ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
$ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1 --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120 --bufferSize 65536
...
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid289.hprof ...
Heap dump file created [1456141975 bytes in 7.760 secs]

Issue Links

• is blocked by: RATIS-726 TimeoutScheduler holds on to the raftClientRequest till it times out even though request succeeds
• relates to: HDDS-2375 Refactor BlockOutputStream to allow flexible buffering