[HIVE-24930] Operator.setDone() short-circuit from child op is not used in vectorized codepath (if childSize == 1)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 24/Mar/21 13:29
• Updated: 17/Nov/22 08:47
• Resolved: 25/Mar/21 17:22

Description

This looks like a possible performance regression in case of limit, considering the following query:

explain vectorization detail select
  ws_item_sk item_sk, d_date,
  sum(ws_sales_price) over (partition by ws_item_sk order by d_date range between 10 preceding and current row) cume_sales,
  last_value(ws_sales_price) over (partition by ws_item_sk order by d_date range between 10 preceding and current row) last_price
from web_sales
    ,date_dim
where ws_sold_date_sk=d_date_sk
  and d_month_seq between 1214 and 1214+11
  and ws_item_sk is not NULL
group by ws_item_sk, d_date, ws_sales_price
limit 100;

in case of vectorized ptf (note: the issue is independent of ptf operator though), the whole pipeline process all the rows, which leads to serious performance regression (note 1439591782 runtime rows for all the operators except limit)

non-vectorized:

set hive.vectorized.execution.ptf.enabled=false;
...
|               Select Operator                      |
|                 Statistics: Num rows: 1415172503/1439591782 Data size: 248969569264 Basic stats: COMPLETE Column stats: COMPLETE |
|                 PTF Operator                       |
|                   Statistics: Num rows: 1415172503/449131 Data size: 248969569264 Basic stats: COMPLETE Column stats: COMPLETE |
|                   Select Operator                  |
|                     Statistics: Num rows: 1415172503/11526 Data size: 565867418560 Basic stats: COMPLETE Column stats: COMPLETE |

vectorized:

set hive.vectorized.execution.ptf.enabled=true;
...
|               Select Operator                      |
|                 Statistics: Num rows: 1415172503/1439591782 Data size: 248969569264 Basic stats: COMPLETE Column stats: COMPLETE |
|                 PTF Operator                       |
|                   Statistics: Num rows: 1415172503/1439591782 Data size: 248969569264 Basic stats: COMPLETE Column stats: COMPLETE |
|                   Select Operator                  |
|                     Statistics: Num rows: 1415172503/1439591782 Data size: 565867418560 Basic stats: COMPLETE Column stats: COMPLETE |
|                       File Output Operator         |
|                         Statistics: Num rows: 100/11300 Data size: 40000 Basic stats: COMPLETE Column stats: COMPLETE |

this is because this short-circuit is missing if childSize==1 (from vectorForward):

      // if all children are done, this operator is also done
      if (childrenDone != 0 && childrenDone == childOperatorsArray.length) {
        setDone(true);
      }

Issue Links

• links to: GitHub Pull Request #2113