[AMBARI-13862] Check status of RS process for AMS in embedded mode

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.2
• Fix Version/s: 2.2.0
• Component/s: ambari-metrics
• Assignee: Dmytro Sen
• Reporter: Dmytro Sen
• Created: 12/Nov/15 16:52
• Updated: 13/Nov/15 21:51
• Resolved: 13/Nov/15 17:31

Description

TimelineMetricStoreWatcher added, it checks TimelineMetricStore functionality by adding and getting a fake metric to/from HBase.

if adding/getting metric fails 3 times in a row, AHS process shuts down with

2015-11-12 15:22:10,142 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.v2.TimelineMetricClusterAggregator: End aggregation cycle @ Thu Nov 12 15:22:10 UTC 2015
2015-11-12 15:22:10,141 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.TimelineMetricClusterAggregatorSecond: Aggregator execution took too long, cancelling sleep. executionTime = 138539
2015-11-12 15:22:10,144 WARN org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.TimelineMetricClusterAggregatorSecond: Last Checkpoint is too old, discarding last checkpoint. lastCheckPointTime = Thu Nov 12 15:16:51 UTC 2015
2015-11-12 15:22:10,144 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.TimelineMetricClusterAggregatorSecond: Saving checkpoint time on first run. Thu Nov 12 15:21:10 UTC 2015
2015-11-12 15:22:13,700 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping phoenix metrics system...
2015-11-12 15:22:13,708 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system stopped.
2015-11-12 15:22:13,709 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system shutdown complete.
2015-11-12 15:22:13,709 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Stopping ApplicationHistory
2015-11-12 15:22:13,710 INFO org.apache.hadoop.ipc.Server: Stopping server on 60200
2015-11-12 15:22:13,716 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60200
2015-11-12 15:22:13,718 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2015-11-12 15:22:13,720 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at c6404.ambari.apache.org/192.168.64.104
************************************************************/

Issue Links

• links to: Review Board