[KYLIN-4194] Throw KylinConfigCannotInitException at STEP "Extract Fact Table Distinct Columns" with spark

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v3.0.0-beta
• Fix Version/s: v3.0.0
• Assignee: Wang Rupeng
• Reporter: Wang Rupeng
• Created: 15/Oct/19 04:00
• Updated: 21/Jan/20 07:48
• Resolved: 29/Oct/19 12:16

Description

When I use spark to extract fact table distinct columns values, kylin throw an exception,

org.apache.kylin.common.KylinConfigCannotInitException: Didn't find KYLIN_CONF or KYLIN_HOME, please set one of them,

see more details in kylin_spark.log ^^ . I tried cdh5.7 and hdp2.4, both platforms got this problem.

It seems like an environment variables issue, but when I set KYLIN_CONF or KYLIN_HOME on each yarn cluster node, it still not work.

As we can see from the log, this exception occurs when TrieDictionaryBuilder try to build  Dictionary<String> objects,  but cannot find KylinConfig using KylinConfig.getInstanceFromEnv() method. 

My solution is to add code in spark job by reset KylinConfig.THREAD_ENV_INSTANCE 

from hdfs metadata as the spark job's metadata is stored in hdfs.And finally it works fine.

Issue Links

• links to: GitHub Pull Request #891
• links to: GitHub Pull Request #891