Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6918

Secure HBase with Kerberos does not work over YARN

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.1, 1.3.0, 1.3.1
    • Fix Version/s: 1.4.0
    • Component/s: YARN
    • Labels:
      None

      Description

      Attempts to access HBase from Spark executors will fail at the auth to the metastore with: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

      This is because HBase Kerberos auth token is not send to the executor. Will need to have something similar to obtainTokensForNamenodes(used for HDFS) in yarn/Client.scala. Storm also needed something similar: https://github.com/apache/storm/pull/226

      I've created a patch for this that required an HBase dependency in the YARN module that we've been using successfully at eBay but am working on a version that does not require the HBase dependency by calling the class loader. Should be ready in a few days.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                deanchen Dean Chen
                Reporter:
                deanchen Dean Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: