Description
Union logical plan is a binary node. However, a typical use case for union is to union a very large number of input sources (DataFrames, RDDs, or files). It is not uncommon to union hundreds of thousands of files. In this case, our optimizer can become very slow due to the large number of logical unions. We should change the Union logical plan to support an arbitrary number of children, and add a single rule in the optimizer (or analyzer?) to collapse all adjacent Unions into one.
Note that this problem doesn't exist in physical plan, because the physical Union already supports arbitrary number of children.
Attachments
Issue Links
- is duplicated by
-
SPARK-12691 Multiple unionAll on Dataframe goes growingly slow.
-
- Resolved
-
- relates to
-
-
- Resolved
-
- links to
