[KYLIN-4904] build cubes error

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: v4.0.0-alpha
• Fix Version/s: v4.0.0-alpha
• Component/s: Query Engine
• Environment: ubuntu20
• Assignee: Unassigned
• Reporter: stayblank
• Created: 07/Feb/21 06:04
• Updated: 19/Feb/21 02:35
• Resolved: 19/Feb/21 02:35

Description

1.不添加  `spark-yarn_2.11-2.4.8-SNAPSHOT.jar` 到  $kylin_home/lib,报这个错
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.kylin.engine.spark.job.NSparkExecutable.runLocalMode(NSparkExecutable.java:389)
at org.apache.kylin.engine.spark.job.NSparkExecutable.doWork(NSparkExecutable.java:153)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:205)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:94)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:205)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Error execute org.apache.kylin.engine.spark.job.ResourceDetectBeforeCubingJob
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:92)
at org.apache.kylin.engine.spark.job.ResourceDetectBeforeCubingJob.main(ResourceDetectBeforeCubingJob.java:100)
... 13 more
Caused by: org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:283)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:89)
... 14 more
 

 

 

2.添加`spark-yarn_2.11-2.4.8-SNAPSHOT.jar` 到 $kylin_home/lib后报这个错
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.kylin.engine.spark.job.NSparkExecutable.runLocalMode(NSparkExecutable.java:389)
at org.apache.kylin.engine.spark.job.NSparkExecutable.doWork(NSparkExecutable.java:153)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:205)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:94)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:205)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ExceptionInInitializerError
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:307)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:126)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:283)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:89)
at org.apache.kylin.engine.spark.job.ResourceDetectBeforeCubingJob.main(ResourceDetectBeforeCubingJob.java:100)
... 13 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.6.7-3
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:67)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:59)
at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
... 24 more