[HIVE-24897] Is null filter on partitioning column leads to non-vectorized execution

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: László Bodor
• Created: 17/Mar/21 14:57
• Updated: 17/Mar/21 15:00

Description

like an ordinary value = 'xy', IS NULL might also work when we filter on the partitioning column

EXPLAIN VECTORIZATION DETAIL select p_mfgr, p_name, p_timestamp, rowindex, p_date, p_retailprice,
count(*) over(partition by p_timestamp) as cs,
sum(p_retailprice) over(partition by p_timestamp) as s
from vector_ptf_part_simple_orc
where p_timestamp IS NULL;

 Reduce Vectorization:                  |
|                 enabled: true                      |
|                 enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true |
|                 notVectorizedReason: PTF operator: Unexpected hive type name void |
|                 vectorized: false                  |
|             Reduce Operator Tree:                  |
|               Select Operator                      |
|                 expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: date), VALUE._col4 (type: double), VALUE._col6 (type: int) |
|                 outputColumnNames: _col0, _col1, _col2, _col4, _col6 |
|                 Statistics: Num rows: 1 Data size: 556 Basic stats: COMPLETE Column stats: COMPLETE |
|                 PTF Operator                       |
|                   Function definitions:            |
|                       Input definition             |
|                         input alias: ptf_0         |
|                         output shape: _col0: string, _col1: string, _col2: date, _col4: double, _col6: int |
|                         type: WINDOWING            |
|                       Windowing table definition   |
|                         input alias: ptf_1         |
|                         name: windowingtablefunction |
|                         order by: null ASC NULLS FIRST |
|                         partition by: CAST( null AS TIMESTAMP) |
|                         raw input shape:           |
|                         window functions:          |
|                             window function definition |
|                               alias: count_window_0 |
|                               name: count          |
|                               window function: GenericUDAFCountEvaluator |
|                               window frame: ROWS PRECEDING(MAX)~FOLLOWING(MAX) |
|                               isStar: true         |
|                             window function definition |
|                               alias: sum_window_1  |
|                               arguments: _col4     |
|                               name: sum            |
|                               window function: GenericUDAFSumDouble |
|                               window frame: ROWS PRECEDING(MAX)~FOLLOWING(MAX) |

the result consists of all p_timestamp=NULL rows:

+-----------------+--------------------------------------------+--------------+-----------+---------+----------------+-----+--------------------+
|     p_mfgr      |                   p_name                   | p_timestamp  | rowindex  | p_date  | p_retailprice  | cs  |         s          |
+-----------------+--------------------------------------------+--------------+-----------+---------+----------------+-----+--------------------+
| Manufacturer#2  | almond aquamarine rose maroon antique      | NULL         | 1         | NULL    | 900.66         | 8   | 7891.070000000001  |
| Manufacturer#5  | almond antique blue firebrick mint         | NULL         | 7         | NULL    | 1789.69        | 8   | 7891.070000000001  |
| Manufacturer#2  | almond antique violet turquoise frosted    | NULL         | 13        | NULL    | 1800.7         | 8   | 7891.070000000001  |
| Manufacturer#3  | almond antique forest lavender goldenrod   | NULL         | 19        | NULL    | 590.27         | 8   | 7891.070000000001  |
| Manufacturer#4  | almond antique gainsboro frosted violet    | NULL         | 25        | NULL    | NULL           | 8   | 7891.070000000001  |
| Manufacturer#1  | almond antique chartreuse lavender yellow  | NULL         | 26        | NULL    | 1753.76        | 8   | 7891.070000000001  |
| Manufacturer#2  | almond aquamarine sandy cyan gainsboro     | NULL         | 32        | NULL    | 1000.6         | 8   | 7891.070000000001  |
| Manufacturer#3  | almond antique metallic orange dim         | NULL         | 38        | NULL    | 55.39          | 8   | 7891.070000000001  |
+-----------------+--------------------------------------------+--------------+-----------+---------+----------------+-----+--------------------+