Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Encountered a case where not closing the Ozone client lead to memory leak related to the usage of OzoneConfiguration objects.
This was added via HDDS-10182
OzoneConfiguration is a bulky object which contains a hashmap having a lot of properties.
In XceiverClientMetrics.java we instantiate the conf object only to fetch one property. If possible this should be removed and passed as a member variable instead.
private OzoneConfiguration conf = new OzoneConfiguration(); private int[] intervals = conf.getInts(OzoneConfigKeys .OZONE_XCEIVER_CLIENT_METRICS_PERCENTILES_INTERVALS_SECONDS_KEY);
From the memory leak report in an app creating multiple unclosed client instances .
"1,663 instances of org.apache.hadoop.hdds.conf.OzoneConfiguration, loaded by com.xxxx.UrlClassLoader @ 0xd05fd5c0 occupy 650,830,032 (61.63%) bytes.
Most of these instances are referenced from one instance of java.util.HashMap$Node[], loaded by <system class loader>, which occupies 69,680 (0.01%) bytes. The instance is referenced by org.apache.hadoop.metrics2.impl.MetricsSystemImpl @ 0xd1453c38, loaded by com.xxxx.UrlClassLoader @ 0xd05fd5c0.
Thread java.util.TimerThread @ 0xd12ff010 Timer for 'XceiverClientMetrics' metrics system has a local variable or reference to org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4 @ 0xd12ff1a8 which is on the shortest path to java.util.HashMap$Node[4096] @ 0xfacaa5f8. The thread java.util.TimerThread @ 0xd12ff010 Timer for 'XceiverClientMetrics' metrics system keeps local variables with total size 464 (0.00%) bytes.
Significant stack frames and local variables
- java.util.TimerThread.mainLoop()V (Timer.java:553)
- org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4 @ 0xd12ff1a8 retains 56 (0.00%) bytes
How does the leak happen?
1. The app keeps on creating new Ozone clients but does not close it
2. Each client object creates an XceiverClientMetrics object that holds a reference to OzoneConfiguration object and registers this metric with the MetricsRegister.
3. Even after the client is done with its task, the conf object is still referenced as the XceiverClientMetrics object is not unregistered and metricsSystem still tracks the metric and holds an active reference to the conf object.
Closing the client will unregister the metric removing the active reference but we should eliminate usage if possible