[RATIS-1743] Memory leak in SegmentedRaftLogWorker due to metrics

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0, 2.4.1
• Fix Version/s: 3.0.0, 2.4.1
• Component/s: server
• Target Version/s: 3.0.0, 2.4.1
• Assignee: Tsz-wo Sze
• Reporter: Attila Doroszlai
• Created: 13/Nov/22 07:48
• Updated: 25/Nov/22 23:00
• Resolved: 16/Nov/22 04:21

Description

OOME happens in Ozone integration tests. Currently Xmx=2g, but increasing it does not help.

https://github.com/adoroszlai/hadoop-ozone/actions/runs/3450185096/jobs/5761108630#step:5:3155

[INFO] Running org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA
Error:  java.lang.OutOfMemoryError: Java heap space
Error:  Tests run: 8, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 426.774 s <<< FAILURE! - in org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA

java.lang.OutOfMemoryError: Java heap space
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.lambda$new$4(SegmentedRaftLogWorker.java:223)
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$$Lambda$603/1771708635.get(Unknown Source)
	at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogOutputStream.write(SegmentedRaftLogOutputStream.java:101)
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$WriteLog.execute(SegmentedRaftLogWorker.java:568)
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.run(SegmentedRaftLogWorker.java:320)
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$$Lambda$595/1626598428.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:750)

Ozone registers JMX reporter (this is not new):

https://github.com/apache/ozone/blob/a13c62b60556cd003ee2149179f72029d9e35756/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/RatisDropwizardExports.java#L51-L53

    MetricRegistries.global()
        .addReporterRegistration(MetricsReporting.jmxReporter(),
            MetricsReporting.stopJmxReporter());

Based on the heap dump and test log, SegmentedRaftLogWorker instances are retained by JmxMBeanServer after close().

The problem is probably not new, but its effect is much worse now, because SegmentedRaftLogWorker recently got a shared buffer (RATIS-1717).

config in Ozone

raft.server.log.appender.buffer.byte-limit = 33554432 (custom)

See screenshot for GC root.

CC Tsz-wo Sze, Song Ziyang

Issue Links

• relates to: RATIS-1717 Perf: Use global serialize buf to avoid temp buf
• relates to: RATIS-1741 Add a removeReporterRegistration method
• links to: GitHub Pull Request #782
• links to: GitHub Pull Request #783
• links to: GitHub Pull Request #784