Minor Description
In case there are a lot of distinct values, count distinct becomes too slow since it tries to hash partial results to one map. It can be improved by creating buckets/partial maps in an intermediate stage where same key from multiple partial maps of first stage hash to the same bucket. Later we can sum the size of these buckets to get total distinct count.
Attachments
Issue Links
- duplicates
-
SPARK-12077 Use more robust plan for single distinct aggregation
-
- Resolved
-
- links to
