[HIVE-20366] TPC-DS query78 stats estimates are off for is null filter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Query Planning
• Target Version/s: 4.0.0
• Assignee: Vineet Garg
• Reporter: Vineet Garg
• Created: 11/Aug/18 04:11
• Updated: 17/Nov/22 08:55
• Resolved: 20/Aug/18 20:26

Description

In Query 78, there is Left outer join between fact table combos: stores_sales LOJ store_returns, catalog_sales LOJ catalog_returns and web_sales LOJ web_returns. Each of these joins estimates only a single row and the result is BROADCAST and causes hash table memory errors

         Reducer 12                                 |
|             Execution mode: vectorized, llap       |
|             Reduce Operator Tree:                  |
+----------------------------------------------------+
|                      Explain                       |
+----------------------------------------------------+
|               Map Join Operator                    |
|                 condition map:                     |
|                      Left Outer Join 0 to 1        |
|                 keys:                              |
|                   0 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1 (type: bigint) |
|                   1 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1 (type: bigint) |
|                 outputColumnNames: _col0, _col1, _col3, _col4, _col5, _col6, _col8 |
|                 input vertices:                    |
|                   1 Map 14                         |
|                 Statistics: Num rows: 10282477384 Data size: 534184867432 Basic stats: COMPLETE Column stats: COMPLETE |
|                 Filter Operator                    |
|                   predicate: _col8 is null (type: boolean) |
|                  * Statistics: Num rows: 1* Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE |

Issue Links

• is related to: HIVE-20636 Improve number of null values estimation after outer join
• links to: ReivewBoard