Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
2.1.2
-
None
Description
STR:
1)Deploy old version (1.6.1)
2)Make Ambari Only Upgrade to 2.1.2.1
3)Enable security
4)Run Tez Check
Actual results:
Check Tez is failed
stderr: /var/lib/ambari-agent/data/errors-878.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py", line 98, in <module> TezServiceCheck().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 216, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py", line 75, in service_check bin_dir = params.hadoop_bin_dir File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/execute_hadoop.py", line 55, in action_run environment = self.resource.environment, File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 260, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'hadoop --config /etc/hadoop/conf jar /usr/lib/tez/tez-mapreduce-examples*.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/' returned 255. 15/10/29 15:07:55 INFO impl.TimelineClientImpl: Timeline service address: http://amb-upg161-rhel6mysql1446082270-4.novalocal:8188/ws/v1/timeline/ 15/10/29 15:07:56 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 15/10/29 15:07:57 INFO client.TezClientUtils: Permissions on staging directory hdfs://nameservice/user/ambari-qa/.staging/application_1446130514848_0008 are incorrect: rwxr-xr-x. Fixing permissions to correct value rwx------ 15/10/29 15:07:57 INFO examples.OrderedWordCount: Creating Tez Session 15/10/29 15:07:57 INFO impl.TimelineClientImpl: Timeline service address: http://amb-upg161-rhel6mysql1446082270-4.novalocal:8188/ws/v1/timeline/ 15/10/29 15:07:58 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 15/10/29 15:07:58 INFO impl.YarnClientImpl: Submitted application application_1446130514848_0008 15/10/29 15:07:58 INFO examples.OrderedWordCount: Created Tez Session 15/10/29 15:07:58 ERROR examples.OrderedWordCount: Error occurred when submitting/running DAGs org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/tezsmokeoutput/ already exists at org.apache.tez.mapreduce.examples.OrderedWordCount.main(OrderedWordCount.java:405) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145) at org.apache.tez.mapreduce.examples.ExampleDriver.main(ExampleDriver.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) 15/10/29 15:07:58 INFO examples.OrderedWordCount: Shutting down session 15/10/29 15:07:58 INFO client.TezSession: Shutting down Tez Session, sessionName=OrderedWordCountSession, applicationId=application_1446130514848_0008 15/10/29 15:07:58 INFO client.TezSession: Could not connect to AM, killing session via YARN, sessionName=OrderedWordCountSession, applicationId=application_1446130514848_0008 15/10/29 15:07:58 INFO impl.YarnClientImpl: Killed application application_1446130514848_0008 org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/tezsmokeoutput/ already exists at org.apache.tez.mapreduce.examples.OrderedWordCount.main(OrderedWordCount.java:405) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145) at org.apache.tez.mapreduce.examples.ExampleDriver.main(ExampleDriver.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) stdout: /var/lib/ambari-agent/data/output-878.txt 2015-10-29 15:07:30,192 - File['/var/lib/ambari-agent/tmp/sample-tez-test'] {'content': 'foo\nbar\nfoo\nbar\nfoo', 'mode': 0755} 2015-10-29 15:07:30,194 - HdfsResource['/tmp/tezsmokeinput'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'default_fs': 'hdfs://nameservice', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'hdfs@EXAMPLE.COM', 'user': 'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir': '/etc/hadoop/conf', 'type': 'directory', 'action': ['create_on_execute']} 2015-10-29 15:07:30,196 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://amb-upg161-rhel6mysql1446082270-3.novalocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp_U530o 2>/tmp/tmpyM0NfU''] {'quiet': False} 2015-10-29 15:07:30,268 - checked_call returned (0, '') 2015-10-29 15:07:30,270 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://amb-upg161-rhel6mysql1446082270-2.novalocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpuDg_Sg 2>/tmp/tmp9oz7xH''] {'quiet': False} 2015-10-29 15:07:30,335 - checked_call returned (0, '') 2015-10-29 15:07:30,336 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://amb-upg161-rhel6mysql1446082270-3.novalocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmphnQloS 2>/tmp/tmpogJnzk''] {'quiet': False} 2015-10-29 15:07:30,396 - checked_call returned (0, '') 2015-10-29 15:07:30,397 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://amb-upg161-rhel6mysql1446082270-2.novalocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpzarBbN 2>/tmp/tmpbCW3Fp''] {'quiet': False} 2015-10-29 15:07:30,462 - checked_call returned (0, '') 2015-10-29 15:07:30,465 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://amb-upg161-rhel6mysql1446082270-2.novalocal:50070/webhdfs/v1/tmp/tezsmokeinput?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp9Zisuq 2>/tmp/tmp9kACst''] {'logoutput': None, 'quiet': False} 2015-10-29 15:07:30,532 - checked_call returned (0, '') 2015-10-29 15:07:30,534 - HdfsResource['/tmp/tezsmokeinput/sample-tez-test'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'source': '/var/lib/ambari-agent/tmp/sample-tez-test', 'default_fs': 'hdfs://nameservice', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'hdfs@EXAMPLE.COM', 'user': 'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir': '/etc/hadoop/conf', 'type': 'file', 'action': ['create_on_execute']} 2015-10-29 15:07:30,535 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://amb-upg161-rhel6mysql1446082270-3.novalocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7Ll4aG 2>/tmp/tmpxNvgA7''] {'quiet': False} 2015-10-29 15:07:30,601 - checked_call returned (0, '') 2015-10-29 15:07:30,603 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://amb-upg161-rhel6mysql1446082270-2.novalocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7VLW9t 2>/tmp/tmpkYktm6''] {'quiet': False} 2015-10-29 15:07:30,669 - checked_call returned (0, '') 2015-10-29 15:07:30,671 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://amb-upg161-rhel6mysql1446082270-3.novalocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpum6gqD 2>/tmp/tmp6TtAHU''] {'quiet': False} 2015-10-29 15:07:30,736 - checked_call returned (0, '') 2015-10-29 15:07:30,737 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://amb-upg161-rhel6mysql1446082270-2.novalocal:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp6u3JvI 2>/tmp/tmpmxbxmj''] {'quiet': False} 2015-10-29 15:07:30,806 - checked_call returned (0, '') 2015-10-29 15:07:30,808 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://amb-upg161-rhel6mysql1446082270-2.novalocal:50070/webhdfs/v1/tmp/tezsmokeinput/sample-tez-test?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpngpNhP 2>/tmp/tmppc8kuv''] {'logoutput': None, 'quiet': False} 2015-10-29 15:07:30,885 - checked_call returned (0, '') 2015-10-29 15:07:30,886 - DFS file /tmp/tezsmokeinput/sample-tez-test is identical to /var/lib/ambari-agent/tmp/sample-tez-test, skipping the copying 2015-10-29 15:07:30,887 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'default_fs': 'hdfs://nameservice', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'hdfs@EXAMPLE.COM', 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/etc/hadoop/conf'} 2015-10-29 15:07:30,888 - ExecuteHadoop['jar /usr/lib/tez/tez-mapreduce-examples*.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/'] {'try_sleep': 5, 'tries': 3, 'bin_dir': '/usr/bin', 'user': 'ambari-qa', 'conf_dir': '/etc/hadoop/conf'} 2015-10-29 15:07:30,890 - Execute['hadoop --config /etc/hadoop/conf jar /usr/lib/tez/tez-mapreduce-examples*.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/'] {'logoutput': None, 'try_sleep': 5, 'environment': {}, 'tries': 3, 'user': 'ambari-qa', 'path': ['/usr/bin']} 2015-10-29 15:07:36,931 - Retrying after 5 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf jar /usr/lib/tez/tez-mapreduce-examples*.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/' returned 255. 15/10/29 15:07:33 INFO impl.TimelineClientImpl: Timeline service address: http://amb-upg161-rhel6mysql1446082270-4.novalocal:8188/ws/v1/timeline/ 15/10/29 15:07:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 15/10/29 15:07:35 INFO client.TezClientUtils: Permissions on staging directory hdfs://nameservice/user/ambari-qa/.staging/application_1446130514848_0006 are incorrect: rwxr-xr-x. Fixing permissions to correct value rwx------ 15/10/29 15:07:35 INFO examples.OrderedWordCount: Creating Tez Session 15/10/29 15:07:35 INFO impl.TimelineClientImpl: Timeline service address: http://amb-upg161-rhel6mysql1446082270-4.novalocal:8188/ws/v1/timeline/ 15/10/29 15:07:36 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 15/10/29 15:07:36 INFO impl.YarnClientImpl: Submitted application application_1446130514848_0006 15/10/29 15:07:36 INFO examples.OrderedWordCount: Created Tez Session 15/10/29 15:07:36 ERROR examples.OrderedWordCount: Error occurred when submitting/running DAGs org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/tezsmokeoutput/ already exists at org.apache.tez.mapreduce.examples.OrderedWordCount.main(OrderedWordCount.java:405) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145) at org.apache.tez.mapreduce.examples.ExampleDriver.main(ExampleDriver.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) 15/10/29 15:07:36 INFO examples.OrderedWordCount: Shutting down session 15/10/29 15:07:36 INFO client.TezSession: Shutting down Tez Session, sessionName=OrderedWordCountSession, applicationId=application_1446130514848_0006 15/10/29 15:07:36 INFO client.TezSession: Could not connect to AM, killing session via YARN, sessionName=OrderedWordCountSession, applicationId=application_1446130514848_0006 15/10/29 15:07:36 INFO impl.YarnClientImpl: Killed application application_1446130514848_0006 org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/tezsmokeoutput/ already exists at org.apache.tez.mapreduce.examples.OrderedWordCount.main(OrderedWordCount.java:405) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145) at org.apache.tez.mapreduce.examples.ExampleDriver.main(ExampleDriver.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) 2015-10-29 15:07:47,842 - Retrying after 5 seconds. Reason: Execution of 'hadoop --config /etc/hadoop/conf jar /usr/lib/tez/tez-mapreduce-examples*.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/' returned 255. 15/10/29 15:07:44 INFO impl.TimelineClientImpl: Timeline service address: http://amb-upg161-rhel6mysql1446082270-4.novalocal:8188/ws/v1/timeline/ 15/10/29 15:07:45 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 15/10/29 15:07:46 INFO client.TezClientUtils: Permissions on staging directory hdfs://nameservice/user/ambari-qa/.staging/application_1446130514848_0007 are incorrect: rwxr-xr-x. Fixing permissions to correct value rwx------ 15/10/29 15:07:46 INFO examples.OrderedWordCount: Creating Tez Session 15/10/29 15:07:46 INFO impl.TimelineClientImpl: Timeline service address: http://amb-upg161-rhel6mysql1446082270-4.novalocal:8188/ws/v1/timeline/ 15/10/29 15:07:47 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 15/10/29 15:07:47 INFO impl.YarnClientImpl: Submitted application application_1446130514848_0007 15/10/29 15:07:47 INFO examples.OrderedWordCount: Created Tez Session 15/10/29 15:07:47 ERROR examples.OrderedWordCount: Error occurred when submitting/running DAGs org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/tezsmokeoutput/ already exists at org.apache.tez.mapreduce.examples.OrderedWordCount.main(OrderedWordCount.java:405) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145) at org.apache.tez.mapreduce.examples.ExampleDriver.main(ExampleDriver.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) 15/10/29 15:07:47 INFO examples.OrderedWordCount: Shutting down session 15/10/29 15:07:47 INFO client.TezSession: Shutting down Tez Session, sessionName=OrderedWordCountSession, applicationId=application_1446130514848_0007 15/10/29 15:07:47 INFO client.TezSession: Could not connect to AM, killing session via YARN, sessionName=OrderedWordCountSession, applicationId=application_1446130514848_0007 15/10/29 15:07:47 INFO impl.YarnClientImpl: Killed application application_1446130514848_0007 org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /tmp/tezsmokeoutput/ already exists at org.apache.tez.mapreduce.examples.OrderedWordCount.main(OrderedWordCount.java:405) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145) at org.apache.tez.mapreduce.examples.ExampleDriver.main(ExampleDriver.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Attachments
Attachments
Issue Links
- links to
