Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
2.2.0
-
None
Description
Prepare Oozie Server fails at upgrade_oozie_database_and_sharelib() step.
Oozie is a special case. In PRE-UPGRADE phase we perform the following steps
1. Stop all Oozie Servers
2. On one Oozie Server execute upgrade_oozie_database_and_sharelib step to create the Oozie DB and Sharelib.
The step upgrade_oozie_database_and_sharelib() invokes
/usr/hdp/{new_version}/oozie/bin/ooziedb.sh upgrade -run
. However, we still haven't executed conf-select, hdp-select which happens in UPGRADE phase at pre_upgrade_restart() step leading to upgrade db step to fail.
2016-02-05 20:32:29,099 - Task. Type: EXECUTE, Script: scripts/oozie_server.py - Function: stop
2016-02-05 20:32:29,331 - In the middle of a stack upgrade/downgrade for Stack HDP and destination version 2.4.0.0-151, determining which hadoop conf dir to use.
2016-02-05 20:32:29,331 - Hadoop conf dir: /usr/hdp/2.4.0.0-151/hadoop/conf
2016-02-05 20:32:29,331 - The hadoop conf dir /usr/hdp/2.4.0.0-151/hadoop/conf exists, will call conf-select on it for version 2.4.0.0-151
2016-02-05 20:32:29,331 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-151/0
2016-02-05 20:32:29,331 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-151 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-02-05 20:32:29,363 - call returned (1, '/etc/hadoop/2.4.0.0-151/0 exist already', '')
2016-02-05 20:32:29,364 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-151 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-02-05 20:32:29,396 - checked_call returned (0, '/usr/hdp/2.4.0.0-151/hadoop/conf -> /etc/hadoop/2.4.0.0-151/0')
2016-02-05 20:32:29,396 - Ensuring that hadoop has the correct symlink structure
2016-02-05 20:32:29,396 - Using hadoop conf dir: /usr/hdp/2.4.0.0-151/hadoop/conf
2016-02-05 20:32:29,400 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh'] {'environment': {'OOZIE_CONFIG': '/usr/hdp/current/oozie-server/conf'}, 'only_if': "ambari-sudo.sh su oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'user': 'oozie'}
2016-02-05 20:32:29,428 - Skipping Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh'] due to only_if
2016-02-05 20:32:29,428 - File['/var/run/oozie/oozie.pid'] {'action': ['delete']}
2016-02-05 20:32:29,442 - Task. Type: EXECUTE, Script: scripts/oozie_server_upgrade.py - Function: upgrade_oozie_database_and_sharelib
2016-02-05 20:32:29,664 - In the middle of a stack upgrade/downgrade for Stack HDP and destination version 2.4.0.0-151, determining which hadoop conf dir to use.
2016-02-05 20:32:29,664 - Hadoop conf dir: /usr/hdp/2.4.0.0-151/hadoop/conf
2016-02-05 20:32:29,664 - The hadoop conf dir /usr/hdp/2.4.0.0-151/hadoop/conf exists, will call conf-select on it for version 2.4.0.0-151
2016-02-05 20:32:29,664 - Checking if need to create versioned conf dir /etc/hadoop/2.4.0.0-151/0
2016-02-05 20:32:29,664 - call['conf-select create-conf-dir --package hadoop --stack-version 2.4.0.0-151 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-02-05 20:32:29,702 - call returned (1, '/etc/hadoop/2.4.0.0-151/0 exist already', '')
2016-02-05 20:32:29,703 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.4.0.0-151 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-02-05 20:32:29,740 - checked_call returned (0, '/usr/hdp/2.4.0.0-151/hadoop/conf -> /etc/hadoop/2.4.0.0-151/0')
2016-02-05 20:32:29,740 - Ensuring that hadoop has the correct symlink structure
2016-02-05 20:32:29,740 - Using hadoop conf dir: /usr/hdp/2.4.0.0-151/hadoop/conf
2016-02-05 20:32:29,744 - Upgrading the Oozie database...
2016-02-05 20:32:29,745 - Execute['/usr/hdp/2.4.0.0-151/oozie/bin/ooziedb.sh upgrade -run'] {'logoutput': True, 'user': 'oozie'}
setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/2.4.0.0-151/oozie/conf}
setting OOZIE_DATA=${OOZIE_DATA:-/var/lib/oozie/data}
setting OOZIE_LOG=${OOZIE_LOG:-/var/log/oozie}
setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/2.4.0.0-151/oozie/oozie-server}
setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
setting CATALINA_PID=${CATALINA_PID:-/var/run/oozie/oozie.pid}
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
Validate DB Connection
20:32:30,786 INFO XLogService:520 -
*******************************************************************************
STARTUP MSG: Oozie BUILD_VERSION [4.2.0.2.4.0.0-151] compiled by [jenkins] on [2016.02.02-03:37:32GMT]
STARTUP MSG: revision [(HEAD@detached]@[git@github.com:hortonworks/oozie.git]
*******************************************************************************
20:32:30,794 INFO XLogService:520 - Log4j configuration file [oozie-log4j.properties]
20:32:30,797 INFO XLogService:520 - Log4j configuration file loaded from [CLASSPATH]
20:32:30,797 INFO XLogService:520 - Log4j reload interval [disabled]
20:32:30,845 WARN XLogUtil:523 - Oozie WS oozie log will be disabled, missing property 'log4j.appender.oozie.File' for 'oozie' appender
20:32:30,845 WARN XLogUtil:523 - Oozie WS oozieError log will be disabled, missing property 'log4j.appender.oozie.File' for 'oozie' appender
20:32:30,845 WARN XLogUtil:523 - Oozie WS oozieaudit log will be disabled, missing property 'log4j.appender.oozie.File' for 'oozie' appender
20:32:30,846 INFO ConfigurationService:520 - Oozie home dir [/usr/hdp/2.4.0.0-151/oozie]
20:32:30,846 INFO ConfigurationService:520 - Oozie conf dir [/usr/hdp/2.4.0.0-151/oozie/conf]
20:32:30,846 INFO ConfigurationService:520 - Oozie conf file [oozie-site.xml]
20:32:31,479 INFO ConfigurationService:520 - Missing site configuration file [/usr/hdp/2.4.0.0-151/oozie/conf/oozie-site.xml]
20:32:31,572 WARN Services:523 - System ID [oozie-oozi] exceeds maximum length [10], trimming
20:32:31,573 INFO Services:520 - Exiting null Entering NORMAL
20:32:31,573 INFO Services:520 - Initialized runtime directory [/tmp/oozie-oozi2862461623458929228.dir]
20:32:31,579 INFO ConfigurationService:520 - Programmatic configuration change, property[oozie.services]=[org.apache.oozie.service.HadoopAccessorService]
20:32:31,580 INFO ConfigurationService:520 - Programmatic configuration change, property[oozie.services.ext]=[]
20:32:31,603 DEBUG Services:526 - Services list obtained from property 'oozie.services'
20:32:31,603 DEBUG Services:526 - Services list obtained from property 'oozie.services.ext'
20:32:31,603 DEBUG Services:526 - Loading service [class org.apache.oozie.service.HadoopAccessorService] implementation [class org.apache.oozie.service.HadoopAccessorService]
20:32:31,610 INFO HadoopAccessorService:520 - JOB_TRACKER_WHITELIST :[], Total entries :0
20:32:31,610 INFO HadoopAccessorService:520 - NAME_NODE_WHITELIST :[], Total entries :0
20:32:31,610 INFO HadoopAccessorService:520 - Oozie Kerberos Authentication [disabled]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-151/oozie/libserver/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-151/oozie/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20:32:32,120 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20:32:32,467 FATAL Services:514 - E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
at org.apache.oozie.service.HadoopAccessorService.parseConfigDirs(HadoopAccessorService.java:231)
at org.apache.oozie.service.HadoopAccessorService.loadHadoopConfigs(HadoopAccessorService.java:247)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:148)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:107)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:177)
at org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:943)
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:951)
at org.apache.oozie.tools.OozieDBCLI.upgradeDB(OozieDBCLI.java:207)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:131)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:79)
20:32:32,469 INFO Services:520 - Shutdown
log4j:WARN No appenders could be found for logger (org.apache.oozie.service.Services).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Error: Could not connect to the database: org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
Stack trace for the error was (for debug purposes):
--------------------------------------
java.lang.Exception: Could not connect to the database: org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:955)
at org.apache.oozie.tools.OozieDBCLI.upgradeDB(OozieDBCLI.java:207)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:131)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:79)
Caused by: org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
at org.apache.oozie.service.HadoopAccessorService.parseConfigDirs(HadoopAccessorService.java:231)
at org.apache.oozie.service.HadoopAccessorService.loadHadoopConfigs(HadoopAccessorService.java:247)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:148)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:107)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:177)
at org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:943)
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:951)
... 3 more
--------------------------------------
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py", line 322, in <module>
OozieUpgrade().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py", line 259, in upgrade_oozie_database_and_sharelib
Execute(database_upgrade_command, user=params.oozie_user, logoutput=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/hdp/2.4.0.0-151/oozie/bin/ooziedb.sh upgrade -run' returned 1. setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/2.4.0.0-151/oozie/conf}
setting OOZIE_DATA=${OOZIE_DATA:-/var/lib/oozie/data}
setting OOZIE_LOG=${OOZIE_LOG:-/var/log/oozie}
setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/2.4.0.0-151/oozie/oozie-server}
setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
setting CATALINA_PID=${CATALINA_PID:-/var/run/oozie/oozie.pid}
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
Validate DB Connection
20:32:30,786 INFO XLogService:520 -
*******************************************************************************
STARTUP MSG: Oozie BUILD_VERSION [4.2.0.2.4.0.0-151] compiled by [jenkins] on [2016.02.02-03:37:32GMT]
STARTUP MSG: revision [(HEAD@detached]@[git@github.com:hortonworks/oozie.git]
*******************************************************************************
20:32:30,794 INFO XLogService:520 - Log4j configuration file [oozie-log4j.properties]
20:32:30,797 INFO XLogService:520 - Log4j configuration file loaded from [CLASSPATH]
20:32:30,797 INFO XLogService:520 - Log4j reload interval [disabled]
20:32:30,845 WARN XLogUtil:523 - Oozie WS oozie log will be disabled, missing property 'log4j.appender.oozie.File' for 'oozie' appender
20:32:30,845 WARN XLogUtil:523 - Oozie WS oozieError log will be disabled, missing property 'log4j.appender.oozie.File' for 'oozie' appender
20:32:30,845 WARN XLogUtil:523 - Oozie WS oozieaudit log will be disabled, missing property 'log4j.appender.oozie.File' for 'oozie' appender
20:32:30,846 INFO ConfigurationService:520 - Oozie home dir [/usr/hdp/2.4.0.0-151/oozie]
20:32:30,846 INFO ConfigurationService:520 - Oozie conf dir [/usr/hdp/2.4.0.0-151/oozie/conf]
20:32:30,846 INFO ConfigurationService:520 - Oozie conf file [oozie-site.xml]
20:32:31,479 INFO ConfigurationService:520 - Missing site configuration file [/usr/hdp/2.4.0.0-151/oozie/conf/oozie-site.xml]
20:32:31,572 WARN Services:523 - System ID [oozie-oozi] exceeds maximum length [10], trimming
20:32:31,573 INFO Services:520 - Exiting null Entering NORMAL
20:32:31,573 INFO Services:520 - Initialized runtime directory [/tmp/oozie-oozi2862461623458929228.dir]
20:32:31,579 INFO ConfigurationService:520 - Programmatic configuration change, property[oozie.services]=[org.apache.oozie.service.HadoopAccessorService]
20:32:31,580 INFO ConfigurationService:520 - Programmatic configuration change, property[oozie.services.ext]=[]
20:32:31,603 DEBUG Services:526 - Services list obtained from property 'oozie.services'
20:32:31,603 DEBUG Services:526 - Services list obtained from property 'oozie.services.ext'
20:32:31,603 DEBUG Services:526 - Loading service [class org.apache.oozie.service.HadoopAccessorService] implementation [class org.apache.oozie.service.HadoopAccessorService]
20:32:31,610 INFO HadoopAccessorService:520 - JOB_TRACKER_WHITELIST :[], Total entries :0
20:32:31,610 INFO HadoopAccessorService:520 - NAME_NODE_WHITELIST :[], Total entries :0
20:32:31,610 INFO HadoopAccessorService:520 - Oozie Kerberos Authentication [disabled]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-151/oozie/libserver/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-151/oozie/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20:32:32,120 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20:32:32,467 FATAL Services:514 - E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
at org.apache.oozie.service.HadoopAccessorService.parseConfigDirs(HadoopAccessorService.java:231)
at org.apache.oozie.service.HadoopAccessorService.loadHadoopConfigs(HadoopAccessorService.java:247)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:148)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:107)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:177)
at org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:943)
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:951)
at org.apache.oozie.tools.OozieDBCLI.upgradeDB(OozieDBCLI.java:207)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:131)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:79)
20:32:32,469 INFO Services:520 - Shutdown
log4j:WARN No appenders could be found for logger (org.apache.oozie.service.Services).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Error: Could not connect to the database: org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
Stack trace for the error was (for debug purposes):
--------------------------------------
java.lang.Exception: Could not connect to the database: org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:955)
at org.apache.oozie.tools.OozieDBCLI.upgradeDB(OozieDBCLI.java:207)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:131)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:79)
Caused by: org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], could not find hadoop configuration directory: /usr/hdp/2.4.0.0-151/oozie/conf/hadoop-conf
at org.apache.oozie.service.HadoopAccessorService.parseConfigDirs(HadoopAccessorService.java:231)
at org.apache.oozie.service.HadoopAccessorService.loadHadoopConfigs(HadoopAccessorService.java:247)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:148)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:107)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:177)
at org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:943)
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:951)
... 3 more
--------------------------------------
Attachments
Attachments
Issue Links
- links to
