[KYLIN-4888] Performance optimization of union query with spark engine

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v4.0.0-alpha
• Fix Version/s: v4.0.0
• Component/s: Spark Engine
• Assignee: Feng Zhu
• Reporter: Feng Zhu
• Created: 27/Jan/21 11:00
• Updated: 08/Feb/21 06:50
• Resolved: 08/Feb/21 06:50

Description

when using union query with spark engine, UnionPlan transforms OLAPUnionRel to spark

DataFrame, when OLAPUnionRel.all = false, distinct transformation of spark will be used, but

it's used in a loop which traversing the DataFrame collection so that we don't have an excepted optimized flattenUnion plan(the CombineUnions rule of spark optimize the distinct, but the nested union plan does not be flattened),there are so many stages in spark dag.  Actuall, distinct transformation should be used only once at last.