[KYLIN-3699] SparkCubingByLayer. Root cause: null

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v2.5.0
• Fix Version/s: v2.6.0
• Component/s: Spark Engine
• Environment: hdp.version： 2.5.0.0-1245 ; kylin: 2.5.0 （apache-kylin-2.5.0-bin-hbase1x）
• Assignee: Chao Long
• Reporter: 风语者
• Created: 23/Nov/18 02:54
• Updated: 12/Jan/19 06:41
• Resolved: 21/Dec/18 11:03

Description

你好：

      我最近在使用 Kylin, 在使用 spark engine 构建 cube 的时候经常会出现一个异常，不知道怎么解决。异常如下：
Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: null
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
at org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(CubeStatsReader.java:297)
at org.apache.kylin.engine.spark.SparkUtil.estimateLayerPartitionNum(SparkUtil.java:108)
at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:182)
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
        我发现，出现这个异常的时候，fact table 与 lookup table 关联后记录条数为0，就会触发这个问题。会在 8 Step Name: Build Cube with Spark 报错。spark 引擎用的 kylin 自带的 spark 2.1.2. 集群环境：hdp.version： 2.5.0.0-1245

       如果用 mapreduce 引擎构建的话，是可以正常构建的，没有数据，流程也可以正常走通。

        不知道这个问题该怎么解决。 

        希望可以得到回复。

        谢谢谢谢。

Issue Links

• is duplicated by: KYLIN-3727 Can not build empty segment, HadoopShellException
• links to: GitHub Pull Request #388
• links to: GitHub Pull Request #391
• links to: GitHub Pull Request #404