[AMBARI-14342] Error during Node Manager start after enabling security post Express Upgrade from 2.1 to 2.3.4

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Assignee: Andrew Onischuk
• Reporter: Andrew Onischuk
• Created: 11/Dec/15 14:08
• Updated: 11/Dec/15 21:12
• Resolved: 11/Dec/15 14:18

Description

Cluster: <http://172.22.111.210:8080/>
Alive for next 24 hours (increase lifetime if required)

*Steps*
Setup cluster at HDP 2.1.15 using Ambari 2.2.0 (unsecure cluster)
Modify relevant tables in DB
Start Express Upgrade to 2.3.4 and let it finish
Enable Kerberos on the cluster

Result:
Observed that while starting services, all Node Manager failed to start and
gave below error:

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 153, in <module>
Nodemanager().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 50, in start
self.configure(env) # FOR SECURITY
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 56, in configure
yarn(name="nodemanager")
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py", line 128, in yarn
content="Marker file to track first start after enabling/disabling security. "
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in _init_
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 87, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/var/lib/hadoop-yarn/nm_security_enabled'] failed, parent directory /var/lib/hadoop-yarn doesn't exist

stdout - attached

Complete artifacts at the time error was seen are here:
<http://linux-jenkins.qe.hortonworks.com/home/jenkins/qe-artifacts/os-s11-3
-jretus-baikaltom20unsecr/ambarieu-e2e-baikaltom20-nosec-noranger-noha-1449839
498/artifacts/screenshots/com.hw.ambari.ui.tests.monitoring.TestAmbariSmokeFun
ctionality/testF_enableSecurity/_11_12_4_42_Exceptions_appeared_during_Enablin
g_Security/>

EU Jenkins job: <http://linux-jenkins.qe.hortonworks.com:8080/job/Run-HDP-
Tests/85136/consoleFull>

Issue Links

• links to: ReviewBoard