Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3874

"Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v2.6.2
    • Component/s: Job Engine
    • Labels:
      None

      Description

      exception

      2019-03-14 16:33:45,113 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
      ...
      Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://hbase-ns-test/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
      at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:116)
      ... 31 more

      hbase-ns-test is the nameservice of HDFS which is used by HBase. 

      In our kylin environment, HBase and MR/HIVE use different HDFS clusters.

      CubeHFileJob.java
      // use current hbase configuration
      Configuration configuration = new Configuration(HBaseConnection.getCurrentHBaseConfiguration());
      merge(configuration, getConf());
      

      "getConf()" contains only the HDFS configuration of MR, and "configuration" contains the HDFS configuration both HBase and MR. See: HBaseConnection#addHBaseClusterNNHAConfiguration()

      So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration to be covered. Mainly parameter "dfs.nameservices"

        Attachments

          Activity

            People

            • Assignee:
              gxcheng Guangxu Cheng
              Reporter:
              gxcheng Guangxu Cheng
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: