-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: v3.0.0-beta
-
Fix Version/s: v3.0.0
-
Component/s: None
-
Labels:None
When I use spark to extract fact table distinct columns values, kylin throw an exception,
org.apache.kylin.common.KylinConfigCannotInitException: Didn't find KYLIN_CONF or KYLIN_HOME, please set one of them,
see more details in kylin_spark.log
^^ . I tried cdh5.7 and hdp2.4, both platforms got this problem.
It seems like an environment variables issue, but when I set KYLIN_CONF or KYLIN_HOME on each yarn cluster node, it still not work.
As we can see from the log, this exception occurs when TrieDictionaryBuilder try to build Dictionary<String> objects, but cannot find KylinConfig using KylinConfig.getInstanceFromEnv() method.
My solution is to add code in spark job by reset KylinConfig.THREAD_ENV_INSTANCE
from hdfs metadata as the spark job's metadata is stored in hdfs.And finally it works fine.
