Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2704

Oozie fails to start if default_realm does not match Oozie prinicpal realm

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 4.3.0
    • Fix Version/s: None
    • Component/s: core, security
    • Labels:
      None
    • Environment:

      CentOS-6.6

      Description

      Problem:

      Oozie fails to start with below exception when default_realm in /etc/krb5.conf does not match with oozie principal realm. (krb5.conf managed by DC Centrify)

      2016-10-06 04:10:15,991 FATAL Services:514 - SERVER[] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/kerberos-2.openstacklocal@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name oozie/kerberos-2.openstacklocal@EXAMPLE.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/kerberos-2.openstacklocal@EXAMPLE.COM
      org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/kerberos-2.openstacklocal@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name oozie/kerberos-2.openstacklocal@EXAMPLE.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/kerberos-2.openstacklocal@EXAMPLE.COM
      	at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:209)
      	at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:136)
      	at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:107)
      	at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
      	at org.apache.oozie.service.Services.setService(Services.java:372)
      	at org.apache.oozie.service.Services.loadServices(Services.java:305)
      	at org.apache.oozie.service.Services.init(Services.java:213)
      	at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:177)
      	at org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:943)
      	at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:951)
      	at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:190)
      	at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:128)
      	at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:79)
      Caused by: java.io.IOException: Login failure for oozie/kerberos-2.openstacklocal@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name oozie/kerberos-2.openstacklocal@EXAMPLE.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/kerberos-2.openstacklocal@EXAMPLE.COM
      	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:976)
      	at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:201)
      	... 12 more
      Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name oozie/kerberos-2.openstacklocal@EXAMPLE.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/kerberos-2.openstacklocal@EXAMPLE.COM
      	at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:202)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
      	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
      	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
      	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
      	at javax.security.auth.login.LoginContext.login(LoginContext.java:588)
      	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:967)
      	... 13 more
      Caused by: java.lang.IllegalArgumentException: Illegal principal name oozie/kerberos-2.openstacklocal@EXAMPLE.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/kerberos-2.openstacklocal@EXAMPLE.COM
      	at org.apache.hadoop.security.User.<init>(User.java:50)
      	at org.apache.hadoop.security.User.<init>(User.java:43)
      	at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:200)
      	... 25 more
      Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to oozie/kerberos-2.openstacklocal@EXAMPLE.COM
      	at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:417)
      	at org.apache.hadoop.security.User.<init>(User.java:48)
      	... 27 more
      2016-10-06 04:10:15,998  INFO Services:520 - SERVER[] Shutdown
      
      

      Steps to Reproduce: Oozie will fail to start if default_realm does not match the principal realm.

      cat /etc/krb5.conf
      libdefaults
      default_realm = CENTRIFY.COM
      Oozie Prinical Name: oozie/kerberos-2.openstacklocal@EXAMPLE.COM

      hadoop.security.auth_to_local has a matching rule "RULE:[2:$1@$0](oozie@EXAMPLE.COM)s/.*/oozie/" configured in core-site.xml but still that is not honored.

      Analysis:

      During Oozie HadoopAccessorService#KerberosInit(), it has to get the short user name from the prinicipal name "oozie/kerberos-2.openstacklocal@EXAMPLE.COM". To get a short user name, hadoop security code does the below things:
      1. Default Rule: check if /etc/krb5.conf default_realm matches the prinicipal realm, then return the service name. This will return Null on our case as default_realm is different (DC centrify case)

      2. Checks the hadoop.security.auth_to_local rules for a matching one. The rules set will be empty always since we are creating a new Configuration object inside HadoopAccessorService#kerberosInit(). The new Configuration properties will be empty as oozie process classpath does not have any configuration directory (Checked printing classpath of Oozie process). All other places except this oozie reads configuration from the configured location by system property -Doozie.config.dir. So this also will return Null as the rules set is empty.

      And so the error message "No rules applied to oozie/kerberos-2.openstacklocal@EXAMPLE.COM" is thrown by Hadoop Security code. Solution is either we need to add configuration directory into classpath of oozie process or new Configuration created has to be populated with the rules from configured location like oozie.config.dir.

      Adding core-site.xml into a WEB_INF/lib jar is the only workaround as of now

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Prabhu Joseph Prabhu Joseph
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: