[KYLIN-3137] Spark cubing without hive-site.xml

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v2.2.0
• Fix Version/s: v2.4.0
• Component/s: Job Engine, Others, Spark Engine
• Labels: cdh cloudera configuration hive
• Assignee: Shao Feng Shi
• Reporter: Ruslan Dautkhanov
• Created: 28/Dec/17 19:00
• Updated: 23/Jun/18 14:56
• Resolved: 01/Jun/18 07:07

Description

Getting following exception while trying to build a cube

java.lang.RuntimeException: Cannot find hive-site.xml in kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you must set kylin.env.hadoop-conf-dir to a dir which contains at least core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml
	at org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117)
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
	at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

I am using Kylin binaries for CDH downloaded from kylin.apache.org.
Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's distribution for Hadoop.

hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf

The other four files:
core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml
can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in /etc/hive/conf .

Would be great to have this adjusted for CDH.