Major Description
Problem:
Currently when using 'Dataset.union.distinct.union.distinct' to union some datasets, Optimizer can't combine all adjacent 'Union' operators into a single 'Union', but it can handle this case when using sql.
For example:
The 'Physical Plan' is shown below:
But using sql:
The 'Physical Plan' is shown below:
Root cause:
When using 'Dataset.union.distinct.union.distinct', the operator is 'Deduplicate(Keys, Union)', but AstBuilder transform sql 'Union' to operator 'Distinct(Union)', the rule 'CombineUnions' in Optimizer only handle 'Distinct(Union)' operator but not Deduplicate(Keys, Union).
