[AMBARI-9237] AMS Service Check fails intermittently due to connection error

• Type: Task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: ambari-metrics, ambari-server
• Epic Link: Ambari Metrics System
• Assignee: Dmytro Sen
• Reporter: Dmytro Sen
• Created: 21/Jan/15 14:51
• Updated: 21/Jan/15 16:57
• Resolved: 21/Jan/15 15:41

Description

Installed a single-host Ambari 2.0.0 cluster with HDP 2.2, and AMS.

The service check failed with the following error,

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMS/0.1.0/package/scripts/service_check.py", line 119, in <module>
    AMSServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 184, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMS/0.1.0/package/scripts/service_check.py", line 55, in service_check
    conn.request("POST", self.AMS_METRICS_POST_URL, metric_json, headers)
  File "/usr/lib64/python2.6/httplib.py", line 914, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
    self.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 720, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
socket.error: [Errno 111] Connection refused

It worked the 2nd time, so may need to increase a timeout value.

stdout:

2015-01-20 21:14:58,002 - AMS service check was started.
2015-01-20 21:14:58,007 - Generated metrics:
{
  "metrics": [
    {
      "metricname": "AMS.SmokeTest.FakeMetric",
      "appid": "amssmoketestfake",
      "hostname": "c6405.ambari.apache.org",
      "timestamp": 1419860001000,
      "starttime": 1419860001000,
      "metrics": {
        "1419860001000": 0.245129358451,
        "1419860002000": 1421788498.0
      }
    }
  ]
}

2015-01-20 21:14:58,007 - Connecting (POST) to c6405.ambari.apache.org:6188/ws/v1/timeline/metrics/

Issue Links

• links to: Review Board