Description
The upper bound of group-by columns row number is to multiply distinct counts of group-by columns. However, column with only null value will cause the output row number to be 0 which is incorrect.
Ex:
col1 (distinct: 2, rowCount 2)
col2 (distinct: 0, rowCount 2)
group by col1, col2
Actual: output rows: 0
Expected: output rows: 2
var outputRows: BigInt = agg.groupingExpressions.foldLeft(BigInt(1))(
(res, expr) => res * childStats.attributeStats(expr.asInstanceOf[Attribute]).distinctCount)
Attachments
Issue Links
- is related to
-
SPARK-27539 Fix inaccurate aggregate outputRows estimation with column containing null values
-
- Resolved
-
- links to