Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18061

Spark Thriftserver needs to create SPNego principal

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.6.1, 2.0.1
    • 2.3.0
    • SQL
    • None

    Description

      Spark Thriftserver when running in HTTP mode with Kerberos enabled gives a 401 authentication error when receiving beeline HTTP request (with end user as kerberos principal). The similar command works with Hive Thriftserver.

      What we find is Hive thriftserver CLI service creates both hive service and SPNego principal when kerberos is enabled whereas Spark Thriftserver
      only creates hive service principal.

      CLIService.java
      if (UserGroupInformation.isSecurityEnabled()) {
            try {
              HiveAuthFactory.loginFromKeytab(hiveConf);
              this.serviceUGI = Utils.getUGI();
            } catch (IOException e) {
              throw new ServiceException("Unable to login to kerberos with given principal/keytab", e);
            } catch (LoginException e) {
              throw new ServiceException("Unable to login to kerberos with given principal/keytab", e);
            }
      
            // Also try creating a UGI object for the SPNego principal
            String principal = hiveConf.getVar(ConfVars.HIVE_SERVER2_SPNEGO_PRINCIPAL);
            String keyTabFile = hiveConf.getVar(ConfVars.HIVE_SERVER2_SPNEGO_KEYTAB);
            if (principal.isEmpty() || keyTabFile.isEmpty()) {
              LOG.info("SPNego httpUGI not created, spNegoPrincipal: " + principal +
                  ", ketabFile: " + keyTabFile);
            } else {
              try {
                this.httpUGI = HiveAuthFactory.loginFromSpnegoKeytabAndReturnUGI(hiveConf);
                LOG.info("SPNego httpUGI successfully created.");
              } catch (IOException e) {
                LOG.warn("SPNego httpUGI creation failed: ", e);
              }
            }
          }
      
      
      SparkSQLCLIService.scala
      if (UserGroupInformation.isSecurityEnabled) {
            try {
              HiveAuthFactory.loginFromKeytab(hiveConf)
              sparkServiceUGI = Utils.getUGI()
              setSuperField(this, "serviceUGI", sparkServiceUGI)
            } catch {
              case e @ (_: IOException | _: LoginException) =>
                throw new ServiceException("Unable to login to kerberos with given principal/keytab", e)
            }
          }
      
      

      The patch will add missing SPNego principal to Spark Thriftserver.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jerryshao Saisai Shao
            cmirashi Chandana Mirashi
            Votes:
            3 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment