Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3699

SparkCubingByLayer. Root cause: null

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v2.5.0
    • Fix Version/s: v2.6.0
    • Component/s: Spark Engine
    • Labels:
      None
    • Environment:
      hdp.version: 2.5.0.0-1245
      kylin: 2.5.0 (apache-kylin-2.5.0-bin-hbase1x)

      Description

      你好:

            我最近在使用 Kylin, 在使用 spark engine 构建 cube 的时候经常会出现一个异常,不知道怎么解决。异常如下:
      Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: null
      at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
      at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.NullPointerException
      at org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(CubeStatsReader.java:297)
      at org.apache.kylin.engine.spark.SparkUtil.estimateLayerPartitionNum(SparkUtil.java:108)
      at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:182)
      at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
              我发现,出现这个异常的时候,fact table 与 lookup table 关联后记录条数为0,就会触发这个问题。会在 8 Step Name: Build Cube with Spark 报错。spark 引擎用的 kylin 自带的 spark 2.1.2. 集群环境:hdp.version: 2.5.0.0-1245

             如果用 mapreduce 引擎构建的话,是可以正常构建的,没有数据,流程也可以正常走通。

              不知道这个问题该怎么解决。 

              希望可以得到回复。

              谢谢谢谢。

        Attachments

        1. 异常步骤.png
          19 kB
          风语者
        2. 异常信息.png
          65 kB
          风语者
        3. Error.png
          9 kB
          风语者
        4. 构建结果.png
          74 kB
          风语者
        5. mapreduce step8.png
          10 kB
          风语者

          Issue Links

            Activity

              People

              • Assignee:
                Wayne0101 Chao Long
                Reporter:
                windTa1ker 风语者
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: