[KYLIN-4922] Failed to build cube

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: v4.0.0-beta
• Fix Version/s: v4.0.0-beta
• Component/s: Spark Engine
• Labels: build
• Environment: kylin v4.0-beta、CDH-6.3.1、Spark version CDH-2.4.0，
• Assignee: Unassigned
• Reporter: YANG LIU
• Created: 02/Mar/21 09:19
• Updated: 14/Oct/21 09:58
• Resolved: 10/Mar/21 01:53

Description

Error Log:

java.io.IOException: OS command error exit with return code: 1, error message: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/spark) overrides detected (/opt/cloudera/parcels/CDH/lib/spark).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" org.apache.spark.SparkException: Cluster deploy mode is not compatible with master "local"
at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:859)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:290)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
The command is:
export HADOOP_CONF_DIR=/opt/kylin/hadoop_conf && /opt/cloudera/parcels/CDH/lib/spark/bin/spark-submit --class org.apache.kylin.engine.spark.application.SparkEntry --conf 'spark.yarn.queue=default' --conf 'spark.history.fs.logDirectory=hdfs:///kylin/spark-history' --conf 'spark.master=local' --conf 'spark.hadoop.yarn.timeline-service.enabled=false' --conf 'spark.driver.cores=1' --conf