[KYLIN-3693] TopN, Count distinct incorrect in Spark engine

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v2.5.0, v2.5.1
• Fix Version/s: v2.5.2
• Component/s: Spark Engine
• Assignee: Shao Feng Shi
• Reporter: Shao Feng Shi
• Created: 18/Nov/18 13:54
• Updated: 04/Dec/18 04:28
• Resolved: 27/Nov/18 06:17

Description

 Long Chao firstly reported this bug; And then we found the currect integration test didn't cover this count distinct, topn measures with Spark engine. In "ITKylinQueryTest", some queries are not tested in the "left" join case, which means they were not covered the spark case. If we manually change to built the "left join" cube with spark, IT will report wrong result:

 

 

[ERROR] Failures: 
[ERROR]   ITCombinationTest>ITKylinQueryTest.testPreciselyDistinctCountQuery:284->KylinTestBase.execAndCompQuery:544->KylinTestBase.execAndCompQuery:647->KylinTestBase.assertTableEquals:691 value (table=query result of query01, row=0, col=USER_COUNT) expected:<8[46]> but was:<8[52]>
[ERROR]   ITCombinationTest>ITKylinQueryTest.testTopNQuery:277->KylinTestBase.execAndCompQuery:544->KylinTestBase.execAndCompQuery:647->KylinTestBase.assertTableEquals:691 value (table=query result of query82, row=0, col=CAL_DT) expected:<2012-01-[28]> but was:<2012-01-[05]>
[INFO] 

 

Issue Links

• is caused by: KYLIN-3472 TopN merge in Spark engine performance tunning
• is duplicated by: KYLIN-3696 TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多