Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1910

Support Separate HBase Cluster with NN HA and Kerberos Authentication

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.5.3
    • Fix Version/s: v1.5.4
    • Component/s: Job Engine
    • Labels:
      None

      Description

      Since KYLIN-957, we've support separate hbase cluster deployment. However, it didn't support hbase cluster with NameNode HA and Kerberos Authentication perfectly.

      The key point is the cube building MR job need to access two hdfs cluster, the main cluster stored source data and the hbase cluster stored the cube result data. The MR task couldn't access hbase cluster with HA qualified path, like hdfs://hbase-cluster:8020/path, because that couldn't found dfs.nameservices related configs of hbase cluster.
      To solve this problem, we need to read the hbase cluster HA configs and set into job conf.

      Another point is about authentication tokens. During job submitting, the Resource Manager try to renew all tokens, to ensure can keep tokens alive. The hdfs token of hbase cluster renewing would cause exception, because that RM couldn't found the hbase cluster HA configs.
      This problem can be resolved with YARN-3021 support.

        Attachments

        1. KYLIN-1910.patch
          6 kB
          Yerui Sun

          Activity

            People

            • Assignee:
              sunyerui Yerui Sun
              Reporter:
              sunyerui Yerui Sun
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: