[AMBARI-25267] Post user creation hook Operation is hanging sometimes until the operation is aborted or Ambari server is restarted

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved

Environment

ambari-2.6.2
HDP-2.6.5

• Assignee: Unassigned
• Reporter: Akhil Naik
• Created: 26/Apr/19 11:09
• Updated: 26/Apr/19 11:16

Description

i have enabled the automatic Creation of user directory for users of ambari (Post user Creation Hook)

When a new user is added in AD and is synced with ambari sometimes the Post User creation Hook operation is hanging infinitely . This happens very rarely and once in two weeks or so . Once hanged i need to abort the operation or restart the ambari-server for the operation to finish.

from logs i can see :

03 Feb 2018 06:00:23,102  INFO [pool-3-thread-1] UserHookService:131 - Preparing hook execution for event: UserCreatedEvent{eventType=USER_CREATED}
03 Feb 2018 06:00:23,117 ERROR [ambari-action-scheduler] ActionScheduler:783 - Execution command has no timeout parameter{"clusterName":"VWFS_ProdKons","requestId":1894,"stageId":-1,"taskId":8497,"commandId":"1894--1","hostname":"_internal_ambari","role":"AMBARI_SERVER_ACTION","hostLevelParams":{},"roleParams":{"ACTION_USER_NAME":"ambari","ACTION_NAME":"org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction"},"roleCommand":"EXECUTE","clusterHostInfo":{},"configurations":{},"configuration_attributes":{},"configurationTags":{},"forceRefreshConfigTagsBeforeExecution":false,"commandParams":{"cmd-hdfs-principal":"[e-mail]-AD","cmd-input-file":"/var/lib/ambari-server/data/tmp/user_hook_input_1517634023107.csv","cluster-security-type":"KERBEROS","cmd-hdfs-user":"hdfs","cmd-payload":"{\"dkx83au\":[\"de-p-ambariusers-kp-s-g\",\"de-adm-hadoopappops-kp-s-g\"],\"dkx8h9n\":[\"de-p-ambariusers-kp-s-g\"],\"dkx8nwd\":[\"de-p-ambariusers-kp-s-g\"]}","cmd-hdfs-keytab":"/etc/security/keytabs/hdfs.headless.keytab","hook-script":"/var/lib/ambari-server/resources/scripts/

then the ambari-server logs fills with Actionscheduler logs till the server is restarted or task is aborted.

call doesn't reach's the 'PostUserCreationHookServerAction' at any stage which happens in my local cluster
restarted ambari-server doesnt help

16 Apr 2019 13:25:08,323  WARN [ambari-action-scheduler] ActionScheduler:782 - Host: _internal_ambari, role: AMBARI_SERVER_ACTION, actionId: 506--1 expired and will be failed
16 Apr 2019 13:25:08,326  INFO [ambari-action-scheduler] ActionScheduler:923 - Service name is , component name is AMBARI_SERVER_ACTIONskipping sending ServiceComponentHostOpFailedEvent for AMBARI_SERVER_ACTION
16 Apr 2019 13:25:08,327  INFO [ambari-action-scheduler] ActionScheduler:809 - Removing command from queue, host=_internal_ambari, commandId=506--1 
16 Apr 2019 13:25:08,327 ERROR [ambari-action-scheduler] ActionScheduler:754 - Execution command has no timeout parameter{“clusterName”:“TESTCLUSTER”,“requestId”:1894,“stageId”:-1,“taskId”:8497,“commandId”:“1894--1”,“hostname”:“_internal_ambari”,“role”:“AMBARI_SERVER_ACTION”,“hostLevelParams”:{},“roleParams”:{“ACTION_USER_NAME”:“ambari”,“ACTION_NAME”:“org.apache.ambari.server.serveraction.users.PostUserCreationHookServerAction”},“roleCommand”:“EXECUTE”,“clusterHostInfo”:{},“configurations”:{},“configuration_attributes”:{},“configurationTags”:{},“forceRefreshConfigTagsBeforeExecution”:false,“commandParams”:{“cmd-hdfs-principal”:“[e-mail]”,“cmd-input-file”:“/var/lib/ambari-server/data/tmp/user_hook_input_1517634023107.csv”,“cluster-security-type”:“KERBEROS”,“cmd-hdfs-user”:“hdfs”,“cmd-payload”:“{\“dqwe3au\“:[\“dere-p-ambariusers-kp-s-g\“,\“de-adm-hadoopappops-kp-s-g\“],\“dkx8123h9n\“:[\“de-p-ambariusers-kp-s-g\“],\“dkx8nwd\“:[\“de-p-ambariusers-kp-s-g\“]}“,”cmd-hdfs-keytab”:“/etc/security/keytabs/hdfs.headless.keytab”,“hook-script”:“/var/lib/ambari-server/resources/scripts/
..
..

Looks like some issue with ambari action scheduler.