[HDDS-2653] Improve executor memory usage in new Freon tests

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 0.5.0
• Component/s: freon
• Labels: pull-request-available
• Target Version/s: 0.5.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 30/Nov/19 14:33
• Updated: 24/Feb/20 20:51
• Resolved: 05/Dec/19 17:20

Description

New Freon tests (descendants of BaseFreonGenerator) suffer from a similar memory issue due to concurrency handling as HDDS-1785.

Steps to reproduce:

export HADOOP_OPTS='-Xmx256M -XX:+HeapDumpOnOutOfMemoryError'
ozone freon omkg -F ONE -n 33554432 -t 10 -p omkg

Freon attempts to submit 32M tasks to the executor, requiring at least 1.5GB memory.

Issue Links

• links to: GitHub Pull Request #284