[AMBARI-21798] Oozie server crashes post migration after regen kerberos keytabs and restart all services

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.5.2
• Fix Version/s: 2.5.2
• Assignee: Robert Levas
• Reporter: Di Li
• Created: 23/Aug/17 19:51
• Updated: 24/Aug/17 15:02
• Resolved: 24/Aug/17 14:26

Description

This is specifically for the IOP *4.2.0 *to HDP migration

Start with an IOP 4.2.0 cluster with both Oozie and Knox installed, enable Kerberos then run thru the migration. Once EU is finalized, regen Kerberos keytabs via Ambari web UI and check the checkbox to make Ambari restart all services as part of the regen process.

Notice Oozie crashes after a successful restart with following error in the oozie.log file

2017-08-23 12:39:48,480 FATAL Services:514 - SERVER[umiak2.fyre.ibm.com] E0550: Could not normalize host name [${host}], Could not resolve host [${host}], ${host}: Name or service not known 
org.apache.oozie.service.ServiceException: E0550: Could not normalize host name [${host}], Could not resolve host [${host}], ${host}: Name or service not known

This is because Knox kerberos.json in IOP 4.2.0 has oozie proxy setting as
"oozie.service.ProxyUserService.proxyuser.knox.hosts": "${hosts}",

In HDP 2.6.2, the same property is now
"oozie.service.ProxyUserService.proxyuser.${knox-env/knox_user}.hosts": "${clusterHostInfo/knox_gateway_hosts}", where the ${hosts} placeholder is changed and Knox params.py no longer has the logic to calculate it .

A manual fix is to change Kerboers config for Knox - update "oozie.service.ProxyUserService.proxyuser.knox.hosts" to have value "${clusterHostInfo/knox_gateway_hosts}" ( done via UI, this will kick off regen keytabs and restart services)

Issue Links

• links to: ReviewBoard