STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Tez DagId: prasanth_20170131172458_d3f3d6c8-3459-4a3c-8659-4f902fc279ca:1 Edges: Reducer 3 <- Map 10 (BROADCAST_EDGE), Map 11 (BROADCAST_EDGE), Map 12 (BROADCAST_EDGE), Map 13 (BROADCAST_EDGE), Map 14 (BROADCAST_EDGE), Map 15 (BROADCAST_EDGE), Map 2 (CUSTOM_SIMPLE_EDGE), Map 7 (CUSTOM_SIMPLE_EDGE), Map 8 (BROADCAST_EDGE), Map 9 (BROADCAST_EDGE) Reducer 4 <- Map 1 (CUSTOM_SIMPLE_EDGE), Reducer 3 (CUSTOM_SIMPLE_EDGE) Reducer 5 <- Reducer 4 (SIMPLE_EDGE) Reducer 6 <- Reducer 5 (SIMPLE_EDGE) DagName: Vertices: Map 1 Map Operator Tree: TableScan alias: catalog_returns filterExpr: cr_item_sk is not null (type: boolean) Statistics: Num rows: 1440033112 Data size: 23040529792 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: cr_item_sk is not null (type: boolean) Statistics: Num rows: 1440033112 Data size: 23040529792 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: cr_item_sk (type: bigint), cr_order_number (type: bigint) outputColumnNames: _col0, _col1 Statistics: Num rows: 1440033112 Data size: 23040529792 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: bigint), _col1 (type: bigint) sort order: ++ Map-reduce partition columns: _col0 (type: bigint), _col1 (type: bigint) Statistics: Num rows: 1440033112 Data size: 23040529792 Basic stats: COMPLETE Column stats: COMPLETE Execution mode: vectorized, llap LLAP IO: all inputs Map 10 Map Operator Tree: TableScan alias: customer_demographics filterExpr: ((cd_marital_status = 'M') and cd_demo_sk is not null) (type: boolean) Statistics: Num rows: 1920800 Data size: 178634400 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: ((cd_marital_status = 'M') and cd_demo_sk is not null) (type: boolean) Statistics: Num rows: 274400 Data size: 25519200 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: cd_demo_sk (type: bigint) outputColumnNames: _col0 Statistics: Num rows: 274400 Data size: 25519200 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: bigint) sort order: + Map-reduce partition columns: _col0 (type: bigint) Statistics: Num rows: 274400 Data size: 25519200 Basic stats: COMPLETE Column stats: COMPLETE Execution mode: vectorized, llap LLAP IO: all inputs Map 11 Map Operator Tree: TableScan alias: household_demographics filterExpr: ((hd_buy_potential = '1001-5000') and hd_demo_sk is not null) (type: boolean) Statistics: Num rows: 7200 Data size: 720000 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: ((hd_buy_potential = '1001-5000') and hd_demo_sk is not null) (type: boolean) Statistics: Num rows: 1440 Data size: 144000 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: hd_demo_sk (type: bigint) outputColumnNames: _col0 Statistics: Num rows: 1440 Data size: 145440 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: bigint) sort order: + Map-reduce partition columns: _col0 (type: bigint) Statistics: Num rows: 1440 Data size: 145440 Basic stats: COMPLETE Column stats: COMPLETE Execution mode: vectorized, llap LLAP IO: all inputs Map 12 Map Operator Tree: TableScan alias: promotion Statistics: Num rows: 2000 Data size: 16000 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: p_promo_sk (type: bigint) outputColumnNames: _col0 Statistics: Num rows: 2000 Data size: 16000 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: bigint) sort order: + Map-reduce partition columns: _col0 (type: bigint) Statistics: Num rows: 2000 Data size: 16000 Basic stats: COMPLETE Column stats: COMPLETE Execution mode: vectorized, llap LLAP IO: all inputs Map 13 Map Operator Tree: TableScan alias: d3 filterExpr: d_date_sk is not null (type: boolean) Statistics: Num rows: 73049 Data size: 584392 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: d_date_sk is not null (type: boolean) Statistics: Num rows: 73049 Data size: 584392 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: d_date_sk (type: bigint) outputColumnNames: _col0 Statistics: Num rows: 73049 Data size: 584392 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: bigint) sort order: + Map-reduce partition columns: _col0 (type: bigint) Statistics: Num rows: 73049 Data size: 584392 Basic stats: COMPLETE Column stats: COMPLETE Execution mode: vectorized, llap LLAP IO: all inputs Map 14 Map Operator Tree: TableScan alias: item filterExpr: i_item_sk is not null (type: boolean) Statistics: Num rows: 402000 Data size: 75576000 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: i_item_sk is not null (type: boolean) Statistics: Num rows: 402000 Data size: 75576000 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: i_item_sk (type: int), i_item_desc (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 402000 Data size: 75576000 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: UDFToLong(_col0) (type: bigint) sort order: + Map-reduce partition columns: UDFToLong(_col0) (type: bigint) Statistics: Num rows: 402000 Data size: 75576000 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col1 (type: string) Execution mode: vectorized, llap LLAP IO: all inputs Map 15 Map Operator Tree: TableScan alias: warehouse filterExpr: w_warehouse_sk is not null (type: boolean) Statistics: Num rows: 25 Data size: 2700 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: w_warehouse_sk is not null (type: boolean) Statistics: Num rows: 25 Data size: 2700 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: w_warehouse_sk (type: bigint), w_warehouse_name (type: string) outputColumnNames: _col0, _col1 Statistics: Num rows: 25 Data size: 2700 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: bigint) sort order: + Map-reduce partition columns: _col0 (type: bigint) Statistics: Num rows: 25 Data size: 2700 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col1 (type: string) Execution mode: vectorized, llap LLAP IO: all inputs Map 2 Map Operator Tree: TableScan alias: catalog_sales filterExpr: (cs_item_sk is not null and cs_bill_cdemo_sk is not null and cs_bill_hdemo_sk is not null and cs_sold_date_sk is not null and cs_ship_date_sk is not null) (type: boolean) Statistics: Num rows: 14399964710 Data size: 861405981696 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (cs_item_sk is not null and cs_bill_cdemo_sk is not null and cs_bill_hdemo_sk is not null and cs_sold_date_sk is not null and cs_ship_date_sk is not null) (type: boolean) Statistics: Num rows: 14185064443 Data size: 848550646340 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: cs_ship_date_sk (type: bigint), cs_bill_cdemo_sk (type: bigint), cs_bill_hdemo_sk (type: bigint), cs_item_sk (type: bigint), cs_promo_sk (type: bigint), cs_order_number (type: bigint), cs_quantity (type: int), cs_sold_date_sk (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7 Statistics: Num rows: 14185064443 Data size: 848550646340 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col3 (type: bigint) sort order: + Map-reduce partition columns: _col3 (type: bigint) Statistics: Num rows: 14185064443 Data size: 848550646340 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col0 (type: bigint), _col1 (type: bigint), _col2 (type: bigint), _col4 (type: bigint), _col5 (type: bigint), _col6 (type: int), _col7 (type: bigint) Execution mode: vectorized, llap LLAP IO: all inputs Map 7 Map Operator Tree: TableScan alias: inventory filterExpr: (inv_item_sk is not null and inv_warehouse_sk is not null) (type: boolean) Statistics: Num rows: 1311525000 Data size: 36460386728 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (inv_item_sk is not null and inv_warehouse_sk is not null) (type: boolean) Statistics: Num rows: 1311525000 Data size: 36460386728 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: inv_item_sk (type: bigint), inv_warehouse_sk (type: bigint), inv_quantity_on_hand (type: int), inv_date_sk (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 1311525000 Data size: 36460386728 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: bigint) sort order: + Map-reduce partition columns: _col0 (type: bigint) Statistics: Num rows: 1311525000 Data size: 36460386728 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col1 (type: bigint), _col2 (type: int), _col3 (type: bigint) Execution mode: vectorized, llap LLAP IO: all inputs Map 8 Map Operator Tree: TableScan alias: d1 filterExpr: ((d_year = 2001) and d_date_sk is not null and d_week_seq is not null) (type: boolean) Statistics: Num rows: 73049 Data size: 1168784 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: ((d_year = 2001) and d_date_sk is not null and d_week_seq is not null) (type: boolean) Statistics: Num rows: 652 Data size: 10432 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: d_date_sk (type: bigint), d_week_seq (type: int) outputColumnNames: _col0, _col1 Statistics: Num rows: 652 Data size: 10432 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: bigint) sort order: + Map-reduce partition columns: _col0 (type: bigint) Statistics: Num rows: 652 Data size: 10432 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col1 (type: int) Select Operator expressions: _col0 (type: bigint) outputColumnNames: _col0 Statistics: Num rows: 652 Data size: 10432 Basic stats: COMPLETE Column stats: COMPLETE Group By Operator keys: _col0 (type: bigint) mode: hash outputColumnNames: _col0 Statistics: Num rows: 326 Data size: 5216 Basic stats: COMPLETE Column stats: COMPLETE Dynamic Partitioning Event Operator Target column: cs_sold_date_sk (bigint) Target Input: catalog_sales Partition key expr: cs_sold_date_sk Statistics: Num rows: 326 Data size: 5216 Basic stats: COMPLETE Column stats: COMPLETE Target Vertex: Map 2 Execution mode: vectorized, llap LLAP IO: all inputs Map 9 Map Operator Tree: TableScan alias: d2 filterExpr: (d_date_sk is not null and d_week_seq is not null) (type: boolean) Statistics: Num rows: 73049 Data size: 876588 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (d_date_sk is not null and d_week_seq is not null) (type: boolean) Statistics: Num rows: 73049 Data size: 876588 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: d_date_sk (type: bigint), d_week_seq (type: int) outputColumnNames: _col0, _col1 Statistics: Num rows: 73049 Data size: 876588 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col1 (type: int), _col0 (type: bigint) sort order: ++ Map-reduce partition columns: _col1 (type: int), _col0 (type: bigint) Statistics: Num rows: 73049 Data size: 876588 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: _col0 (type: bigint) outputColumnNames: _col0 Statistics: Num rows: 73049 Data size: 876588 Basic stats: COMPLETE Column stats: COMPLETE Group By Operator keys: _col0 (type: bigint) mode: hash outputColumnNames: _col0 Statistics: Num rows: 73049 Data size: 876588 Basic stats: COMPLETE Column stats: COMPLETE Dynamic Partitioning Event Operator Target column: inv_date_sk (bigint) Target Input: inventory Partition key expr: inv_date_sk Statistics: Num rows: 73049 Data size: 876588 Basic stats: COMPLETE Column stats: COMPLETE Target Vertex: Map 7 Execution mode: vectorized, llap LLAP IO: all inputs Reducer 3 Execution mode: vectorized, llap Reduce Operator Tree: Map Join Operator condition map: Inner Join 0 to 1 keys: 0 KEY.reducesinkkey0 (type: bigint) 1 KEY.reducesinkkey0 (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col9, _col10, _col11 input vertices: 1 Map 7 Statistics: Num rows: 42971168591721 Data size: 3437693487337680 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (_col10 < _col6) (type: boolean) Statistics: Num rows: 14323722863907 Data size: 1145897829112560 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: _col0 (type: bigint), _col1 (type: bigint), _col11 (type: bigint), _col2 (type: bigint), _col3 (type: bigint), _col4 (type: bigint), _col5 (type: bigint), _col7 (type: bigint), _col9 (type: bigint) outputColumnNames: _col0, _col1, _col11, _col2, _col3, _col4, _col5, _col7, _col9 Statistics: Num rows: 14323722863907 Data size: 1145897829112560 Basic stats: COMPLETE Column stats: COMPLETE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col7 (type: bigint) 1 _col0 (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col9, _col11, _col13 input vertices: 1 Map 8 Statistics: Num rows: 5078270778876 Data size: 345322412963568 Basic stats: COMPLETE Column stats: COMPLETE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col13 (type: int), _col11 (type: bigint) 1 _col1 (type: int), _col0 (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col9, _col13 input vertices: 1 Map 9 Statistics: Num rows: 431730146 Data size: 25903808760 Basic stats: COMPLETE Column stats: COMPLETE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col1 (type: bigint) 1 _col0 (type: bigint) outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col9, _col13 input vertices: 1 Map 10 Statistics: Num rows: 61675738 Data size: 3207138376 Basic stats: COMPLETE Column stats: COMPLETE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col2 (type: bigint) 1 _col0 (type: bigint) outputColumnNames: _col0, _col3, _col4, _col5, _col9, _col13 input vertices: 1 Map 11 Statistics: Num rows: 12335148 Data size: 542746512 Basic stats: COMPLETE Column stats: COMPLETE Map Join Operator condition map: Left Outer Join0 to 1 keys: 0 _col4 (type: bigint) 1 _col0 (type: bigint) outputColumnNames: _col0, _col3, _col5, _col9, _col13, _col21 input vertices: 1 Map 12 Statistics: Num rows: 12335148 Data size: 542746512 Basic stats: COMPLETE Column stats: COMPLETE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col0 (type: bigint) 1 _col0 (type: bigint) outputColumnNames: _col3, _col5, _col9, _col13, _col21 input vertices: 1 Map 13 Statistics: Num rows: 12335148 Data size: 444065328 Basic stats: COMPLETE Column stats: COMPLETE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col3 (type: bigint) 1 UDFToLong(_col0) (type: bigint) outputColumnNames: _col3, _col5, _col9, _col13, _col21, _col24 input vertices: 1 Map 14 Statistics: Num rows: 12335148 Data size: 2713732560 Basic stats: COMPLETE Column stats: COMPLETE Map Join Operator condition map: Inner Join 0 to 1 keys: 0 _col9 (type: bigint) 1 _col0 (type: bigint) outputColumnNames: _col3, _col5, _col13, _col21, _col24, _col26 input vertices: 1 Map 15 Statistics: Num rows: 12335148 Data size: 3848566176 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: _col26 (type: string), _col24 (type: string), _col13 (type: int), _col21 (type: bigint), _col3 (type: bigint), _col5 (type: bigint) outputColumnNames: _col13, _col15, _col21, _col26, _col3, _col5 Statistics: Num rows: 12335148 Data size: 3848566176 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col3 (type: bigint), _col5 (type: bigint) sort order: ++ Map-reduce partition columns: _col3 (type: bigint), _col5 (type: bigint) Statistics: Num rows: 12335148 Data size: 3848566176 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col13 (type: string), _col15 (type: string), _col21 (type: int), _col26 (type: bigint) Reducer 4 Execution mode: vectorized, llap Reduce Operator Tree: Map Join Operator condition map: Right Outer Join0 to 1 keys: 0 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1 (type: bigint) 1 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1 (type: bigint) outputColumnNames: _col15, _col17, _col23, _col28 input vertices: 0 Map 1 Statistics: Num rows: 12335148 Data size: 3651203808 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: _col17 (type: string), _col15 (type: string), _col23 (type: int), CASE WHEN (_col28 is null) THEN (1) ELSE (0) END (type: int), CASE WHEN (_col28 is not null) THEN (1) ELSE (0) END (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4 Statistics: Num rows: 12335148 Data size: 3651203808 Basic stats: COMPLETE Column stats: COMPLETE Group By Operator aggregations: count(_col3), count(_col4), count() keys: _col0 (type: string), _col1 (type: string), _col2 (type: int) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 Statistics: Num rows: 12335148 Data size: 3848566176 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: int) sort order: +++ Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: int) Statistics: Num rows: 12335148 Data size: 3848566176 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col3 (type: bigint), _col4 (type: bigint), _col5 (type: bigint) Reducer 5 Execution mode: vectorized, llap Reduce Operator Tree: Group By Operator aggregations: count(VALUE._col0), count(VALUE._col1), count(VALUE._col2) keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: int) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 Statistics: Num rows: 7116921 Data size: 2220479352 Basic stats: COMPLETE Column stats: COMPLETE Reduce Output Operator key expressions: _col5 (type: bigint), _col0 (type: string), _col1 (type: string), _col2 (type: int) sort order: -+++ Statistics: Num rows: 7116921 Data size: 2220479352 Basic stats: COMPLETE Column stats: COMPLETE TopN Hash Memory Usage: 0.04 value expressions: _col3 (type: bigint), _col4 (type: bigint) Reducer 6 Execution mode: vectorized, llap Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey3 (type: int), VALUE._col0 (type: bigint), VALUE._col1 (type: bigint), KEY.reducesinkkey0 (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 Statistics: Num rows: 7116921 Data size: 2220479352 Basic stats: COMPLETE Column stats: COMPLETE Limit Number of rows: 100 Statistics: Num rows: 100 Data size: 31200 Basic stats: COMPLETE Column stats: COMPLETE File Output Operator compressed: false Statistics: Num rows: 100 Data size: 31200 Basic stats: COMPLETE Column stats: COMPLETE table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: 100 Processor Tree: ListSink