[AMBARI-17256] Yarn service check fails after Ambari upgrade from 2.0.2 to 2.4.0.0 - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.4.0
Component/s: None
Labels:
None

Description

Build # - 2.4.0.0-665.x86_64

Live cluster - <http://172.22.108.220:8080/>

*Steps*

1. Deploy HDP-2.2.6 cluster with Ambari 2.0.2 (secure, HA cluster)
2. Upgrade Ambari to 2.4.0.0
3. Stop and start all services
4. Run Yarn service check

*Result*

stderr: /var/lib/ambari-agent/data/errors-1081.txt

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py", line 152, in <module>
ServiceCheck().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 257, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py", line 140, in service_check
raise Exception("Could not get json response from YARN API")
Exception: Could not get json response from YARN API
stdout: /var/lib/ambari-agent/data/output-1081.txt

2016-06-15 09:03:01,363 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-06-15 09:03:01,365 - call['ambari-python-wrap /usr/bin/hdp-select status hadoop-yarn-resourcemanager']

{'timeout': 20}

2016-06-15 09:03:01,402 - call returned (0, 'hadoop-yarn-resourcemanager - 2.2.6.0-2800')
2016-06-15 09:03:01,406 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-06-15 09:03:01,413 - HdfsResource['/user/ambari-qa']

{'security_enabled': True, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'dfs_type': '', 'default_fs': 'hdfs://nameservice', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'hdfs@EXAMPLE.COM', 'user': 'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/tmp', u'/app-logs', u'/mr-history/done', u'/apps/falcon'], 'mode': 0770}

2016-06-15 09:03:01,417 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM']

{'user': 'hdfs'}

2016-06-15 09:03:01,598 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl --negotiate -u : -s '"'"'http://os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-1.openstacklocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpgeb77n 2>/tmp/tmpfiGN9F'']

{'quiet': False}

2016-06-15 09:03:01,647 - call returned (0, '')
2016-06-15 09:03:01,649 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl --negotiate -u : -s '"'"'http://os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-2.openstacklocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpv9dJHa 2>/tmp/tmpAPWvMO'']

{'quiet': False}

2016-06-15 09:03:01,712 - call returned (0, '')
2016-06-15 09:03:01,713 - NameNode HA states: active_namenodes = [('nn2', 'os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-2.openstacklocal:50070')], standby_namenodes = [('nn1', 'os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-1.openstacklocal:50070')], unknown_namenodes = []
2016-06-15 09:03:01,715 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl --negotiate -u : -s '"'"'http://os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-1.openstacklocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpQ65QjZ 2>/tmp/tmpge44K9'']

{'quiet': False}

2016-06-15 09:03:01,781 - call returned (0, '')
2016-06-15 09:03:01,783 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl --negotiate -u : -s '"'"'http://os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-2.openstacklocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpxvKBS1 2>/tmp/tmpJ_3Fej'']

{'quiet': False}

2016-06-15 09:03:01,828 - call returned (0, '')
2016-06-15 09:03:01,829 - NameNode HA states: active_namenodes = [('nn2', 'os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-2.openstacklocal:50070')], standby_namenodes = [('nn1', 'os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-1.openstacklocal:50070')], unknown_namenodes = []
2016-06-15 09:03:01,830 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%

{http_code}

'"'"' -X GET --negotiate -u : '"'"'http://os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-2.openstacklocal:50070/webhdfs/v1/user/ambari-qa?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpxA6GiR 2>/tmp/tmp4f6kV4'']

{'logoutput': None, 'quiet': False}

2016-06-15 09:03:01,885 - call returned (0, '')
2016-06-15 09:03:01,887 - checked_call['/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa@EXAMPLE.COM; yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000 --queue default']

{'path': '/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin', 'user': 'ambari-qa'}

2016-06-15 09:03:48,567 - checked_call returned (0, '16/06/15 09:03:05 INFO impl.TimelineClientImpl: Timeline service address: http://os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-5.openstacklocal:8188/ws/v1/timeline/\n16/06/15 09:03:05 INFO distributedshell.Client: Initializing Client\n16/06/15 09:03:05 INFO distributedshell.Client: Running Client\n16/06/15 09:03:06 INFO distributedshell.Client: Got Cluster metric info from ASM, numNodeManagers=5\n16/06/15 09:03:06 INFO distributedshell.Client: Got Cluster node info from ASM\n16/06/15 09:03:06 INFO distributedshell.Client: Got node report from ASM for, nodeId=os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-3.openstacklocal:45454, nodeAddressos-r6-hoiqv-nat-s-c224toermlevelsecha-r6-3.openstacklocal:8042, nodeRackName/default-rack, nodeNumContainers0\n16/06/15 09:03:06 INFO distributedshell.Client: Got node report from ASM for, nodeId=os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-2.openstacklocal:45454, nodeAddressos-r6-hoiqv-nat-s-c224toermlevelsecha-r6-2.openstacklocal:8042, nodeRackName/default-rack, nodeNumContainers0\n16/06/15 09:03:06 INFO distributedshell.Client: Got node report from ASM for, nodeId=os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-5.openstacklocal:45454, nodeAddressos-r6-hoiqv-nat-s-c224toermlevelsecha-r6-5.openstacklocal:8042, nodeRackName/default-rack, nodeNumContainers0\n16/06/15 09:03:06 INFO distributedshell.Client: Got node report from ASM for, nodeId=os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-6.openstacklocal:45454, nodeAddressos-r6-hoiqv-nat-s-c224toermlevelsecha-r6-6.openstacklocal:8042, nodeRackName/default-rack, nodeNumContainers0\n16/06/15 09:03:06 INFO distributedshell.Client: Got node report from ASM for, nodeId=os-r6-hoiqv-nat-s-c224toermlevelsecha-r6-1.openstacklocal:45454, nodeAddressos-r6-hoiqv-nat-s-c224toermlevelsecha-r6-1.openstacklocal:8042, nodeRackName/default-rack, nodeNumContainers0\n16/06/15 09:03:06 INFO distributedshell.Client: Queue info, queueName=default, queueCurrentCapacity=0.0, queueMaxCapacity=1.0, queueApplicationCount=0, queueChildQueueCount=0\n16/06/15 09:03:06 INFO distributedshell.Client: User ACL Info for Queue, queueName=root, userAcl=SUBMIT_APPLICATIONS\n16/06/15 09:03:06 INFO distributedshell.Client: User ACL Info for Queue, queueName=root, userAcl=ADMINISTER_QUEUE\n16/06/15 09:03:06 INFO distributedshell.Client: User ACL Info for Queue, queueName=default, userAcl=SUBMIT_APPLICATIONS\n16/06/15 09:03:06 INFO distributedshell.Client: User ACL Info for Queue, queueName=default, userAcl=ADMINISTER_QUEUE\n16/06/15 09:03:06 INFO distributedshell.Client: Max mem capabililty of resources in this cluster 10240\n16/06/15 09:03:06 INFO distributedshell.Client: Max virtual cores capabililty of resources in this cluster 8\n16/06/15 09:03:06 INFO distributedshell.Client: Copy App Master jar from local filesystem and add to local environment\n16/06/15 09:03:08 INFO distributedshell.Client: Set the environment for the application master\n16/06/15 09:03:08 INFO distributedshell.Client: Setting up app master command\n16/06/15 09:03:08 INFO distributedshell.Client: Completed setting up app master command JAVA_HOME/bin/java -Xmx10m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --container_vcores 1 --num_containers 1 --priority 0 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr \n16/06/15 09:03:08 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 19 for ambari-qa on ha-hdfs:nameservice\n16/06/15 09:03:08 INFO distributedshell.Client: Got dt for hdfs://nameservice; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice, Ident: (HDFS_DELEGATION_TOKEN token 19 for ambari-qa)\n16/06/15 09:03:08 INFO distributedshell.Client: Submitting application to ASM\n16/06/15 09:03:09 INFO impl.YarnClientImpl: Submitted application application_1465953261376_0004\n16/06/15 09:03:10 INFO distributedshell.Client: Got application report from ASM for, appId=4, clientToAMToken=Token