Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.2.1
-
None
Description
Steps
- Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, multiple oozie servers)
- Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
- Perform Downgrade back to original version and let it complete
- Run service check for Oozie
Result
Service check failed with error:
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf /usr/hdp/current/hadoop-client/bin ambari-qa True /etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM' returned 1. /usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM; source /usr/hdp/current/oozie-client/conf/oozie-env.sh ; /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job -oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 404, message: Not Found Invalid sub-command: Missing argument for option: info use 'help [sub-command]' for help details Invalid sub-command: Missing argument for option: info use 'help [sub-command]' for help details workflow_status=
catalina.out prints below
Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext resourcesStart SEVERE: Error starting static Resources java.lang.IllegalArgumentException: Document base /usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is not a readable directory at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142) at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324) Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader loadClass INFO: Illegal access: this web application instance has been stopped already. Could not load com.google.common.cache.RemovalCause. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact. java.lang.IllegalStateException at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
Attached complete logs
Note - I did a restart of Oozie service and the issue is no longer seen
Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got created as part of Oozie service restart
ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/ total 136M lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs -> /usr/hdp/2.3.4.0-3485/oozie/doc lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 -> /usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2 -rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war *drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
Running the prepare-war command multiple times causes it to remove the /usr/hdp/current/oozie-server/oozie-server/webapps/oozie folder.
Further, there's a bug that the /usr/hdp/current/oozie-server/.prepare_war_cmd marker file is still generated even if the prepare war command was not ran!
Attachments
Attachments
Issue Links
- links to
