[KYLIN-5208] kylin 4.0.1 构建cube 2个表1对多关系 ，报错 DUP key found ,key=venue_id

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Component/s: Spark Engine
• Assignee: Unassigned
• Reporter: huang song
• Created: 08/Jul/22 09:20
• Updated: 11/Jul/22 08:59

Description

2022-07-08 10:44:56,379 INFO  [pool-1-thread-1] cluster.SchedulerInfoCmdHelper : stderr                                  Dload  Upload   Total   Spent    Left  Speed
2022-07-08 10:44:56,379 INFO  [pool-1-thread-1] cluster.SchedulerInfoCmdHelper : stderr 
================================================================
 at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:308)
 at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:94)
 ... 4 more

}
RetryInfo

{     overrideConf : \{spark.executor.memory=6143MB, spark.executor.memoryOverhead=1228MB}

,
    throwable : java.lang.RuntimeException: Error execute org.apache.kylin.engine.spark.job.CubeBuildJob
 at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:97)
 at org.apache.spark.application.JobWorker$$anon$2.run(JobWorker.scala:55)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Failed to build lookup table B_VENUE_OPEN_TIME snapshot for Dup key found, key= VENUE_ID
 at org.apache.kylin.engine.spark.builder.CubeSnapshotBuilder.$anonfun$checkDupKey$1(CubeSnapshotBuilder.scala:200)
 at org.apache.kylin.engine.spark.builder.CubeSnapshotBuilder.$anonfun$checkDupKey$1$adapted(CubeSnapshotBuilder.scala:190)
 at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
 at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
 at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
 at org.apache.kylin.engine.spark.builder.CubeSnapshotBuilder.checkDupKey(CubeSnapshotBuilder.scala:190)
 at org.apache.kylin.engine.spark.job.ParentSourceChooser.decideFlatTableSource(ParentSourceChooser.scala:83)
 at org.apache.kylin.engine.spark.job.ParentSourceChooser.$anonfun$decideSources$1(ParentSourceChooser.scala:71)