[AMBARI-15100] OOM on TimelineMetricCache in Nodemanager

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.2
• Component/s: ambari-metrics
• Assignee: Dmytro Sen
• Reporter: Siddharth Wagle
• Created: 18/Feb/16 21:53
• Updated: 05/Aug/16 14:33
• Resolved: 26/Feb/16 07:28

Description

Reported on ambari user group:

2016-02-16 18:04:24,510 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException

2016-02-16 18:04:34,509 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException

2016-02-16 18:04:44,510 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException

2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error.  Shutting down now...

java.lang.OutOfMemoryError: Java heap space

       at java.util.HashMap.resize(HashMap.java:703)

       at java.util.HashMap.putVal(HashMap.java:662)

       at java.util.HashMap.put(HashMap.java:611)

       at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)

       at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)

       at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)

       at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)

       at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)

       at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)

       at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)

       at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)

       at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)

2016-02-16 18:04:59,433 INFO  util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException

Issue Links

• links to: Review Board