[CALCITE-4351] RelMdUtil#numDistinctVals always returns 0 for large inputs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.26.0
• Fix Version/s: 1.33.0
• Component/s: core
• Labels: pull-request-available
• Assignee: Liya Fan
• Reporter: Caizhi Weng
• Created: 23/Oct/20 07:59
• Updated: 07/Feb/23 22:33
• Resolved: 29/Dec/22 21:33

Description

Previous implementation of RelMdUtil#numDistinctVals uses the approximation ln(1 + x) ~= x when x is small.

However CALCITE-4132 remove this approximation to make the result more accurate. This causes the function to calculate an incorrect result for large inputs (for example, when domainSize = 1e18 and numSelected = 1e10 the result is 0) due to precision problems.

What I would suggest is to treat small and large inputs in different ways. For small inputs we use the new, more precise function and for large inputs we use the old, approximated function.

Issue Links

• causes: FLINK-19780 FlinkRelMdDistinctRowCount#getDistinctRowCount(Calc) will always return 0 when number of rows are large
• causes: FLINK-21946 FlinkRelMdUtil.numDistinctVals produces exceptional Double.NaN result when domainSize is in range(0,1)
• is caused by: CALCITE-4132 Estimate the number of distinct values more accurately
• is duplicated by: CALCITE-5431 Method RelMdUtil$numDistinctVals() wrongly return zero if the method input domainSize is a very large value
• relates to: CALCITE-5431 Method RelMdUtil$numDistinctVals() wrongly return zero if the method input domainSize is a very large value
• links to: GitHub Pull Request #3015