Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4194

Throw KylinConfigCannotInitException at STEP "Extract Fact Table Distinct Columns" with spark

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v3.0.0-beta
    • Fix Version/s: v3.0.0
    • Component/s: None
    • Labels:
      None

      Description

      When I use spark to extract fact table distinct columns values, kylin throw an exception,

      org.apache.kylin.common.KylinConfigCannotInitException: Didn't find KYLIN_CONF or KYLIN_HOME, please set one of them,

      see more details in kylin_spark.log ^^ . I tried cdh5.7 and hdp2.4, both platforms got this problem.

      It seems like an environment variables issue, but when I set KYLIN_CONF or KYLIN_HOME on each yarn cluster node, it still not work.

      As we can see from the log, this exception occurs when TrieDictionaryBuilder try to build  Dictionary<String> objects,  but cannot find KylinConfig using KylinConfig.getInstanceFromEnv() method. 

      My solution is to add code in spark job by reset KylinConfig.THREAD_ENV_INSTANCE 

      from hdfs metadata as the spark job's metadata is stored in hdfs.And finally it works fine.

        Attachments

        1. kylin_spark.log
          50 kB
          Wang Rupeng
        2. image-2019-10-16-16-13-00-190.png
          253 kB
          Wang Rupeng

          Activity

            People

            • Assignee:
              wangrupeng Wang Rupeng
              Reporter:
              wangrupeng Wang Rupeng
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: