[AMBARI-14847] Concurrent kinit Commands Cause Alerts To Randomly Trigger

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.2.2
• Component/s: ambari-agent
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 29/Jan/16 18:57
• Updated: 01/Feb/16 19:22
• Resolved: 01/Feb/16 15:12

Description

The alerts framework on each Ambari Agent runs alerts in a threadpool when the job triggers. This can cause the following error to randomly appear and the alert to go CRITICAL:

 Connection failed to http://nat-rare-21-dvitiiuk-2-5.novalocal:8088 (Execution of '/usr/bin/kinit -l 5m -c /var/lib/ambari-agent/tmp/web_alert_cc_f3f99363c3b7d1667f1287ce3a35aa52 -kt /etc/security/keytabs/spnego.service.keytab HTTP/[e-mail] > /dev/null' returned 1.

kinit: Internal credentials cache error while storing credentials while getting initial credentials)

The alerts would randomly go CRITICAL at the end of their ticket expiration time only to become OK again shortly after.

The cause is that the kinit command being executed to create new credentials cannot be run concurrently for the same user.

Issue Links

• links to: Reviewboard