[OOZIE-2704] Oozie fails to start if default_realm does not match Oozie prinicpal realm

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 4.3.0
• Component/s: core, security

Environment

CentOS-6.6

• Assignee: Unassigned
• Reporter: Prabhu Joseph
• Created: 08/Oct/16 05:54
• Updated: 01/Feb/19 00:23

Description

Problem:

Oozie fails to start with below exception when default_realm in /etc/krb5.conf does not match with oozie principal realm. (krb5.conf managed by DC Centrify)

2016-10-06 04:10:15,991 FATAL Services:514 - SERVER[] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/[e-mail] from keytab /etc/security/keytabs/oozie.service.keytab: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name oozie/[e-mail]: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/[e-mail]
org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/[e-mail] from keytab /etc/security/keytabs/oozie.service.keytab: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name oozie/[e-mail]: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/[e-mail]
	at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:209)
	at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:136)
	at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:107)
	at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
	at org.apache.oozie.service.Services.setService(Services.java:372)
	at org.apache.oozie.service.Services.loadServices(Services.java:305)
	at org.apache.oozie.service.Services.init(Services.java:213)
	at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:177)
	at org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:943)
	at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:951)
	at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:190)
	at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:128)
	at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:79)
Caused by: java.io.IOException: Login failure for oozie/[e-mail] from keytab /etc/security/keytabs/oozie.service.keytab: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name oozie/[e-mail]: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/[e-mail]
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:976)
	at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:201)
	... 12 more
Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name oozie/[e-mail]: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/[e-mail]
	at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:202)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:588)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:967)
	... 13 more
Caused by: java.lang.IllegalArgumentException: Illegal principal name oozie/[e-mail]: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/[e-mail]
	at org.apache.hadoop.security.User.<init>(User.java:50)
	at org.apache.hadoop.security.User.<init>(User.java:43)
	at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:200)
	... 25 more
Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/[e-mail]
	at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:417)
	at org.apache.hadoop.security.User.<init>(User.java:48)
	... 27 more
2016-10-06 04:10:15,998  INFO Services:520 - SERVER[] Shutdown

Steps to Reproduce: Oozie will fail to start if default_realm does not match the principal realm.

cat /etc/krb5.conf
libdefaults
default_realm = CENTRIFY.COM
Oozie Prinical Name: oozie/[e-mail]

hadoop.security.auth_to_local has a matching rule "RULE:[2:$1@$0]([e-mail])s/.*/oozie/" configured in core-site.xml but still that is not honored.

Analysis:

During Oozie HadoopAccessorService#KerberosInit(), it has to get the short user name from the prinicipal name "oozie/[e-mail]". To get a short user name, hadoop security code does the below things:
1. Default Rule: check if /etc/krb5.conf default_realm matches the prinicipal realm, then return the service name. This will return Null on our case as default_realm is different (DC centrify case)

2. Checks the hadoop.security.auth_to_local rules for a matching one. The rules set will be empty always since we are creating a new Configuration object inside HadoopAccessorService#kerberosInit(). The new Configuration properties will be empty as oozie process classpath does not have any configuration directory (Checked printing classpath of Oozie process). All other places except this oozie reads configuration from the configured location by system property -Doozie.config.dir. So this also will return Null as the rules set is empty.

And so the error message "No rules applied to oozie/[e-mail]" is thrown by Hadoop Security code. Solution is either we need to add configuration directory into classpath of oozie process or new Configuration created has to be populated with the rules from configured location like oozie.config.dir.

Adding core-site.xml into a WEB_INF/lib jar is the only workaround as of now