Query (id=c64ec374919042a8:f359a10f00000000): Summary: Session ID: fb48618e111511a2:cc29f70ed3ce638b Session Type: BEESWAX Start Time: 2017-08-14 21:20:34.442599000 End Time: 2017-08-14 21:43:02.562804000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.10.0-SNAPSHOT RELEASE (build d5b0c6b9366bc54801ac0dc74c942cdbda213b67) User: tarmstrong Connected User: tarmstrong Delegated User: Network Address: ::ffff:127.0.0.1:59064 Default Db: tpch_60_parquet Sql Statement: select count(*) from (select distinct * from lineitem) v Coordinator: tarmstrong-box2:22000 Query Options (non default): MEM_LIMIT=10737418240,MT_DOP=0 Plan: ---------------- Per-Host Resource Reservation: Memory=264.00MB Per-Host Resource Estimates: Memory=197.50GB F02:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 06:AGGREGATE [FINALIZE] | output: count:merge(*) | mem-estimate=10.00MB mem-reservation=0B | tuple-ids=3 row-size=8B cardinality=1 | 05:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=3 row-size=8B cardinality=1 | F01:PLAN FRAGMENT [HASH(tpch_60_parquet.lineitem.l_orderkey,tpch_60_parquet.lineitem.l_partkey,tpch_60_parquet.lineitem.l_suppkey,tpch_60_parquet.lineitem.l_linenumber,tpch_60_parquet.lineitem.l_quantity,tpch_60_parquet.lineitem.l_extendedprice,tpch_60_parquet.lineitem.l_discount,tpch_60_parquet.lineitem.l_tax,tpch_60_parquet.lineitem.l_returnflag,tpch_60_parquet.lineitem.l_linestatus,tpch_60_parquet.lineitem.l_shipdate,tpch_60_parquet.lineitem.l_commitdate,tpch_60_parquet.lineitem.l_receiptdate,tpch_60_parquet.lineitem.l_shipinstruct,tpch_60_parquet.lineitem.l_shipmode,tpch_60_parquet.lineitem.l_comment)] hosts=1 instances=1 Per-Host Resources: mem-estimate=98.40GB mem-reservation=264.00MB 02:AGGREGATE | output: count(*) | mem-estimate=10.00MB mem-reservation=0B | tuple-ids=3 row-size=8B cardinality=1 | 04:AGGREGATE [FINALIZE] | group by: tpch_60_parquet.lineitem.l_orderkey, tpch_60_parquet.lineitem.l_partkey, tpch_60_parquet.lineitem.l_suppkey, tpch_60_parquet.lineitem.l_linenumber, tpch_60_parquet.lineitem.l_quantity, tpch_60_parquet.lineitem.l_extendedprice, tpch_60_parquet.lineitem.l_discount, tpch_60_parquet.lineitem.l_tax, tpch_60_parquet.lineitem.l_returnflag, tpch_60_parquet.lineitem.l_linestatus, tpch_60_parquet.lineitem.l_shipdate, tpch_60_parquet.lineitem.l_commitdate, tpch_60_parquet.lineitem.l_receiptdate, tpch_60_parquet.lineitem.l_shipinstruct, tpch_60_parquet.lineitem.l_shipmode, tpch_60_parquet.lineitem.l_comment | mem-estimate=98.39GB mem-reservation=264.00MB | tuple-ids=1 row-size=267B cardinality=360011594 | 03:EXCHANGE [HASH(tpch_60_parquet.lineitem.l_orderkey,tpch_60_parquet.lineitem.l_partkey,tpch_60_parquet.lineitem.l_suppkey,tpch_60_parquet.lineitem.l_linenumber,tpch_60_parquet.lineitem.l_quantity,tpch_60_parquet.lineitem.l_extendedprice,tpch_60_parquet.lineitem.l_discount,tpch_60_parquet.lineitem.l_tax,tpch_60_parquet.lineitem.l_returnflag,tpch_60_parquet.lineitem.l_linestatus,tpch_60_parquet.lineitem.l_shipdate,tpch_60_parquet.lineitem.l_commitdate,tpch_60_parquet.lineitem.l_receiptdate,tpch_60_parquet.lineitem.l_shipinstruct,tpch_60_parquet.lineitem.l_shipmode,tpch_60_parquet.lineitem.l_comment)] | mem-estimate=0B mem-reservation=0B | tuple-ids=1 row-size=267B cardinality=360011594 | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 Per-Host Resources: mem-estimate=99.08GB mem-reservation=0B 01:AGGREGATE [STREAMING] | group by: tpch_60_parquet.lineitem.l_orderkey, tpch_60_parquet.lineitem.l_partkey, tpch_60_parquet.lineitem.l_suppkey, tpch_60_parquet.lineitem.l_linenumber, tpch_60_parquet.lineitem.l_quantity, tpch_60_parquet.lineitem.l_extendedprice, tpch_60_parquet.lineitem.l_discount, tpch_60_parquet.lineitem.l_tax, tpch_60_parquet.lineitem.l_returnflag, tpch_60_parquet.lineitem.l_linestatus, tpch_60_parquet.lineitem.l_shipdate, tpch_60_parquet.lineitem.l_commitdate, tpch_60_parquet.lineitem.l_receiptdate, tpch_60_parquet.lineitem.l_shipinstruct, tpch_60_parquet.lineitem.l_shipmode, tpch_60_parquet.lineitem.l_comment | mem-estimate=98.39GB mem-reservation=0B | tuple-ids=1 row-size=267B cardinality=360011594 | 00:SCAN HDFS [tpch_60_parquet.lineitem, RANDOM] partitions=1/1 files=51 size=12.56GB stats-rows=360011594 extrapolated-rows=disabled table stats: rows=360011594 size=12.56GB column stats: all mem-estimate=704.00MB mem-reservation=0B tuple-ids=0 row-size=267B cardinality=360011594 ---------------- Estimated Per-Host Mem: 212059697970 Per-Host Memory Reservation: 276824064 Request Pool: default-pool Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail --------------------------------------------------------------------------------------------------------------------------- 06:AGGREGATE 1 450.889us 450.889us 1 1 20.00 KB 10.00 MB FINALIZE 05:EXCHANGE 1 6.351us 6.351us 1 1 0 0 UNPARTITIONED 02:AGGREGATE 1 187.322ms 187.322ms 1 1 16.00 KB 10.00 MB 04:AGGREGATE 1 18m12s 18m12s 360.01M 360.01M 7.74 GB 98.39 GB FINALIZE 03:EXCHANGE 1 4s931ms 4s931ms 360.01M 360.01M 0 0 HASH(tpch_60_parquet.lineit... 01:AGGREGATE 1 2m1s 2m1s 360.01M 360.01M 284.31 MB 98.39 GB STREAMING 00:SCAN HDFS 1 2s293ms 2s293ms 360.01M 360.01M 1.56 GB 704.00 MB tpch_60_parquet.lineitem Errors: Planner Timeline: 5s465ms - Metadata load started: 17.740ms (17.740ms) - Metadata load finished: 5s317ms (5s299ms) - Analysis finished: 5s359ms (41.740ms) - Equivalence classes computed: 5s377ms (18.166ms) - Single node plan created: 5s407ms (30.269ms) - Runtime filters computed: 5s409ms (2.069ms) - Distributed plan created: 5s414ms (4.198ms) - Planning finished: 5s465ms (51.425ms) Query Timeline: 22m28s - Query submitted: 70.709us (70.709us) - Planning finished: 5s475ms (5s475ms) - Submit for admission: 5s475ms (122.736us) - Completed admission: 5s475ms (54.641us) - Ready to start on 1 backends: 5s475ms (74.614us) - All 1 execution backends (3 fragment instances) started: 5s476ms (675.589us) - Rows available: 22m28s (22m22s) - First row fetched: 22m28s (70.320ms) - Unregister query: 22m28s (4.788ms) - ComputeScanRangeAssignmentTimer: 36.921us ImpalaServer: - ClientFetchWaitTimer: 72.655ms - RowMaterializationTimer: 2.411ms Execution Profile c64ec374919042a8:f359a10f00000000:(Total: 22m22s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 0 Filter routing table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- Backend startup latencies: Count: 1, min / max: 0 / 0, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 0, 99.9th %-ile: 0 Per Node Peak Memory Usage: tarmstrong-box2:22000(9.21 GB) - FiltersReceived: 0 (0) - FinalizationTimer: 0.000ns Averaged Fragment F02:(Total: 22m22s, non-child: 71.301ms, % non-child: 0.01%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:22m22s max:22m22s mean: 22m22s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 28.01 KB (28680) - PerHostPeakMemUsage: 9.21 GB (9892828404) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 22m22s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 22m22s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 4 (4) Fragment Instance Lifecycle Timings: - ExecTime: 70.998ms - ExecTreeExecTime: 1.307us - OpenTime: 22m22s - ExecTreeOpenTime: 22m22s - PrepareTime: 10.529ms - ExecTreePrepareTime: 33.713us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 11.65K (11647) - BlocksRecycled: 11.65K (11648) - BufferedPins: 219 (219) - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 8.00 GB (8589934592) - PeakMemoryUsage: 8.00 GB (8589934592) - ScratchBytesRead: 82.91 GB (89028296704) - ScratchBytesWritten: 82.91 GB (89028296704) - ScratchFileUsedBytes: 82.91 GB (89028296704) - ScratchReads: 10.61K (10613) - ScratchWrites: 10.61K (10613) - TotalBufferWaitTime: 2m10s - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 8m12s PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 10.218ms, non-child: 10.218ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 9.906ms AGGREGATION_NODE (id=6):(Total: 22m22s, non-child: 450.889us, % non-child: 0.00%) - BuildTime: 2.134us - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 20.00 KB (20480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) EXCHANGE_NODE (id=5):(Total: 22m22s, non-child: 22m22s, % non-child: 100.00%) - BytesReceived: 16.00 B (16) - ConvertRowBatchTime: 482.000ns - DeserializeRowBatchTimer: 1.869us - FirstBatchArrivalWaitTime: 22m22s - PeakMemoryUsage: 0 - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Coordinator Fragment F02: Instance c64ec374919042a8:f359a10f00000000 (host=tarmstrong-box2:22000):(Total: 22m22s, non-child: 71.301ms, % non-child: 0.01%) MemoryUsage(32s000ms): 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 28.01 KB (28680) - PerHostPeakMemUsage: 9.21 GB (9892828404) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 22m22s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 22m22s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 4 (4) Fragment Instance Lifecycle Timings: - ExecTime: 70.998ms - ExecTreeExecTime: 1.307us - OpenTime: 22m22s - ExecTreeOpenTime: 22m22s - PrepareTime: 10.529ms - ExecTreePrepareTime: 33.713us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 11.65K (11647) - BlocksRecycled: 11.65K (11648) - BufferedPins: 219 (219) - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 8.00 GB (8589934592) - PeakMemoryUsage: 8.00 GB (8589934592) - ScratchBytesRead: 82.91 GB (89028296704) - ScratchBytesWritten: 82.91 GB (89028296704) - ScratchFileUsedBytes: 82.91 GB (89028296704) - ScratchReads: 10.61K (10613) - ScratchWrites: 10.61K (10613) - TotalBufferWaitTime: 2m10s - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 8m12s PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 10.218ms, non-child: 10.218ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 9.906ms AGGREGATION_NODE (id=6):(Total: 22m22s, non-child: 450.889us, % non-child: 0.00%) - BuildTime: 2.134us - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 20.00 KB (20480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) EXCHANGE_NODE (id=5):(Total: 22m22s, non-child: 6.351us, % non-child: 0.00%) BytesReceived(32s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 16.00 B (16) - ConvertRowBatchTime: 482.000ns - DeserializeRowBatchTimer: 1.869us - FirstBatchArrivalWaitTime: 22m22s - PeakMemoryUsage: 0 - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F01:(Total: 22m22s, non-child: 2.510ms, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:22m22s max:22m22s mean: 22m22s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 1.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 7.77 GB (8340401406) - PerHostPeakMemUsage: 9.21 GB (9892828404) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 4m6s - TotalNetworkSendTime: 274.394us - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 17.60K (17596) - TotalThreadsTotalWallClockTime: 22m22s - TotalThreadsSysTime: 13s800ms - TotalThreadsUserTime: 7m44s - TotalThreadsVoluntaryContextSwitches: 346.33K (346327) Fragment Instance Lifecycle Timings: - ExecTime: 451.459us - ExecTreeExecTime: 1.031us - OpenTime: 22m22s - ExecTreeOpenTime: 22m22s - PrepareTime: 18.889ms - ExecTreePrepareTime: 98.663us DataStreamSender (dst_id=5):(Total: 50.079us, non-child: 50.079us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 435.05 KB/sec - OverallThroughput: 312.01 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - RowsReturned: 1 (1) - SerializeBatchTime: 4.005us - TransmitDataRPCTime: 35.915us - UncompressedRowBatchSize: 16.00 B (16) CodeGen:(Total: 180.677ms, non-child: 180.677ms, % non-child: 100.00%) - CodegenTime: 2.920ms - CompileTime: 51.868ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 65 (65) - NumInstructions: 1.93K (1925) - OptimizationTime: 111.589ms - PeakMemoryUsage: 962.50 KB (985600) - PrepareTime: 16.797ms AGGREGATION_NODE (id=2):(Total: 22m22s, non-child: 187.322ms, % non-child: 0.01%) - BuildTime: 119.890ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 16.00 KB (16384) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) AGGREGATION_NODE (id=4):(Total: 22m22s, non-child: 18m12s, % non-child: 81.41%) - BuildTime: 3m29s - GetNewBlockTime: 2m10s - GetResultsTime: 4s966ms - HTResizeTime: 13s270ms - HashBuckets: 536.87M (536870912) - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 1 (1) - NumRepartitions: 15 (15) - PartitionsCreated: 256 (256) - PeakMemoryUsage: 7.74 GB (8315709067) - PinTime: 8m13s - RowsRepartitioned: 337.51M (337512336) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 268.23 K/sec - SpilledPartitions: 15 (15) - UnpinTime: 5.964ms EXCHANGE_NODE (id=3):(Total: 4m9s, non-child: 4m9s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42862325692) - ConvertRowBatchTime: 2s808ms - DeserializeRowBatchTimer: 1m12s - FirstBatchArrivalWaitTime: 687.058ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.44 M/sec - SendersBlockedTimer: 1m55s - SendersBlockedTotalTimer(*): 1m55s Fragment F01: Instance c64ec374919042a8:f359a10f00000002 (host=tarmstrong-box2:22000):(Total: 22m22s, non-child: 2.510ms, % non-child: 0.00%) MemoryUsage(32s000ms): 4.39 GB, 7.38 GB, 7.16 GB, 7.02 GB, 6.49 GB, 6.80 GB, 5.92 GB, 6.64 GB, 4.13 GB, 4.51 GB, 4.90 GB, 5.29 GB, 5.66 GB, 6.04 GB, 3.90 GB, 4.36 GB, 1.85 GB, 3.77 GB, 2.62 GB, 3.73 GB, 3.02 GB, 3.26 GB, 3.32 GB, 2.89 GB, 3.79 GB, 2.72 GB, 4.14 GB, 2.00 GB, 4.23 GB, 1.89 GB, 3.75 GB, 2.86 GB, 3.34 GB, 3.24 GB, 2.89 GB, 3.62 GB, 3.13 GB, 3.59 GB, 2.67 GB, 4.31 GB, 1.85 GB ThreadUsage(32s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 7.77 GB (8340401406) - PerHostPeakMemUsage: 9.21 GB (9892828404) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 4m6s - TotalNetworkSendTime: 274.394us - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 17.60K (17596) - TotalThreadsTotalWallClockTime: 22m22s - TotalThreadsSysTime: 13s800ms - TotalThreadsUserTime: 7m44s - TotalThreadsVoluntaryContextSwitches: 346.33K (346327) Fragment Instance Lifecycle Timings: - ExecTime: 451.459us - ExecTreeExecTime: 1.031us - OpenTime: 22m22s - ExecTreeOpenTime: 22m22s - PrepareTime: 18.889ms - ExecTreePrepareTime: 98.663us DataStreamSender (dst_id=5):(Total: 50.079us, non-child: 50.079us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 435.05 KB/sec - OverallThroughput: 312.01 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - RowsReturned: 1 (1) - SerializeBatchTime: 4.005us - TransmitDataRPCTime: 35.915us - UncompressedRowBatchSize: 16.00 B (16) CodeGen:(Total: 180.677ms, non-child: 180.677ms, % non-child: 100.00%) - CodegenTime: 2.920ms - CompileTime: 51.868ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 65 (65) - NumInstructions: 1.93K (1925) - OptimizationTime: 111.589ms - PeakMemoryUsage: 962.50 KB (985600) - PrepareTime: 16.797ms AGGREGATION_NODE (id=2):(Total: 22m22s, non-child: 187.322ms, % non-child: 0.01%) ExecOption: Codegen Enabled - BuildTime: 119.890ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 16.00 KB (16384) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) AGGREGATION_NODE (id=4):(Total: 22m22s, non-child: 18m12s, % non-child: 81.41%) ExecOption: Codegen Enabled, Spilled - BuildTime: 3m29s - GetNewBlockTime: 2m10s - GetResultsTime: 4s966ms - HTResizeTime: 13s270ms - HashBuckets: 536.87M (536870912) - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 1 (1) - NumRepartitions: 15 (15) - PartitionsCreated: 256 (256) - PeakMemoryUsage: 7.74 GB (8315709067) - PinTime: 8m13s - RowsRepartitioned: 337.51M (337512336) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 268.23 K/sec - SpilledPartitions: 15 (15) - UnpinTime: 5.964ms EXCHANGE_NODE (id=3):(Total: 4m9s, non-child: 4s931ms, % non-child: 1.98%) BytesReceived(8s000ms): 387.39 MB, 1.32 GB, 2.28 GB, 3.19 GB, 4.01 GB, 4.65 GB, 5.21 GB, 5.84 GB, 6.53 GB, 7.32 GB, 7.97 GB, 8.72 GB, 9.36 GB, 10.05 GB, 10.77 GB, 11.45 GB, 12.04 GB, 12.68 GB, 13.34 GB, 13.98 GB, 14.59 GB, 15.19 GB, 15.86 GB, 16.54 GB, 17.31 GB, 17.98 GB, 18.67 GB, 19.44 GB, 20.21 GB, 20.92 GB, 21.46 GB, 22.14 GB, 22.82 GB, 23.53 GB, 24.30 GB, 24.95 GB, 25.66 GB, 26.28 GB, 26.91 GB, 27.60 GB, 28.22 GB, 29.07 GB, 29.74 GB, 30.39 GB, 31.12 GB, 31.81 GB, 32.47 GB, 33.10 GB, 33.79 GB, 34.40 GB, 35.16 GB, 35.84 GB, 36.42 GB, 37.07 GB, 37.76 GB, 38.60 GB, 39.25 GB - BytesReceived: 39.92 GB (42862325692) - ConvertRowBatchTime: 2s808ms - DeserializeRowBatchTimer: 1m12s - FirstBatchArrivalWaitTime: 687.058ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.44 M/sec - SendersBlockedTimer: 1m55s - SendersBlockedTotalTimer(*): 1m55s Averaged Fragment F00:(Total: 7m39s, non-child: 1m45s, % non-child: 22.99%) split sizes: min: 12.56 GB, max: 12.56 GB, avg: 12.56 GB, stddev: 0 completion times: min:7m39s max:7m39s mean: 7m39s stddev:0.000ns execution rates: min:27.97 MB/sec max:27.97 MB/sec mean:27.97 MB/sec stddev:0.11 B/sec num instances: 1 - AverageThreadTokens: 5.30 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.84 GB (1974069265) - PerHostPeakMemUsage: 9.21 GB (9892828404) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 1m56s - TotalStorageWaitTime: 34s040ms - TotalThreadsInvoluntaryContextSwitches: 8.77K (8768) -09 GB, 990.84 MB, 858.19 MB, 642.42 MB, 1.07 GB, 1006.14 MB, 907.90 MB, 858.38 MB, 925.77 MB, 1.07 GB, 967.22 MB, 862.82 MB, 844.15 MB, 923.12 MB, 1.04 GB, 910.24 MB, 894.13 MB, 770.60 MB, 1.09 GB, 980.07 MB, 907.09 MB, 886.43 MB, 856.27 MB, 1.07 GB, 941.15 MB, 932.99 MB, 828.17 MB, 920.96 MB, 1.03 GB, 863.03 MB, 675.39 MB ThreadUsage(8s000ms): 8, 8, 8, 8, 8, 8, 6, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4 - AverageThreadTokens: 5.30 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.84 GB (1974069265) - PerHostPeakMemUsage: 9.21 GB (9892828404) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 1m56s - TotalStorageWaitTime: 34s040ms - TotalThreadsInvoluntaryContextSwitches: 8.77K (8768) - TotalThreadsTotalWallClockTime: 40m37s - TotalThreadsSysTime: 2s156ms - TotalThreadsUserTime: 7m6s - TotalThreadsVoluntaryContextSwitches: 352.99K (352992) Fragment Instance Lifecycle Timings: - ExecTime: 7m39s - ExecTreeExecTime: 2m3s - OpenTime: 167.131ms - ExecTreeOpenTime: 80.968us - PrepareTime: 34.346ms - ExecTreePrepareTime: 17.975ms DataStreamSender (dst_id=3):(Total: 3m50s, non-child: 3m50s, % non-child: 100.00%) - BytesSent: 39.92 GB (42862325692) - NetworkThroughput(*): 3.35 GB/sec - OverallThroughput: 177.59 MB/sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 3m38s - TransmitDataRPCTime: 11s923ms - UncompressedRowBatchSize: 91.46 GB (98207171234) CodeGen:(Total: 181.581ms, non-child: 181.581ms, % non-child: 100.00%) - CodegenTime: 2.186ms - CompileTime: 54.179ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 85 (85) - NumInstructions: 2.44K (2444) - OptimizationTime: 112.410ms - PeakMemoryUsage: 1.19 MB (1251328) - PrepareTime: 14.690ms AGGREGATION_NODE (id=1):(Total: 2m3s, non-child: 2m1s, % non-child: 98.15%) ExecOption: Streaming Preaggregation, Codegen Enabled - GetNewBlockTime: 782.079us - GetResultsTime: 9.444ms - HTResizeTime: 13.892ms - HashBuckets: 1.05M (1048576) - LargestPartitionPercent: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 284.31 MB (298118539) - PinTime: 0.000ns - ReductionFactorEstimate: 1.00 - ReductionFactorThresholdToExpand: 2.00 - RowsPassedThrough: 359.23M (359225162) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.91 M/sec - StreamingTime: 2m - UnpinTime: 0.000ns HDFS_SCAN_NODE (id=0):(Total: 2s293ms, non-child: 2s293ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 51 out of 51 Hdfs Read Thread Concurrency Bucket: 0:84.42% 1:7.081% 2:2.179% 3:0.2179% 4:1.198% 5:0.4357% 6:4.466% File Formats: PARQUET/SNAPPY:816 BytesRead(8s000ms): 1017.65 MB, 1.49 GB, 1.54 GB, 1.63 GB, 1.71 GB, 1.73 GB, 1.82 GB, 2.55 GB, 2.58 GB,