During scale testing of ozone with 350k+ containers and nearly 1 million replica reports it was observed that, there is a sudden burst in SCM heap usage . In HDFS, the full block report interval is 6 hours by default and in between, there are incremental block reports. Similarly, there are incremental reports in SCM . Setting the full container report interval to 1 hour make things quite stable as determined from tests and 60s for full report seems very aggressive.
1.Increase default container report interval to 60 mins from 60 sec currently .
2. Increase pipeline report interval