Query (id=7d4ad07a20dbeeed:6f4174f00000000): Summary: Session ID: f84755fd23448d06:ed727993551443af Session Type: BEESWAX Start Time: 2017-08-17 18:56:25.202088000 End Time: 2017-08-17 19:41:49.690863000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.10.0-SNAPSHOT RELEASE (build 473f717d0f38397f61f35e0ac9e32226d837a28a) User: tarmstrong Connected User: tarmstrong Delegated User: Network Address: ::ffff:127.0.0.1:55002 Default Db: tpch_60_parquet Sql Statement: select * from lineitem l1 join lineitem l2 on l1.l_orderkey = l2.l_orderkey and l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey and l1.l_linenumber = l2.l_linenumber where l1.l_tax < l2.l_tax Coordinator: tarmstrong-box2:22000 Query Options (non default): MEM_LIMIT=10737418240,MT_DOP=0,DEFAULT_SPILLABLE_BUFFER_SIZE=8388608 Plan: ---------------- Per-Host Resource Reservation: Memory=136.00MB Per-Host Resource Estimates: Memory=99.77GB F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=0B mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 05:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=0,1 row-size=534B cardinality=360011594 | F02:PLAN FRAGMENT [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] hosts=1 instances=1 Per-Host Resources: mem-estimate=98.39GB mem-reservation=136.00MB 02:HASH JOIN [INNER JOIN, PARTITIONED] | hash predicates: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | fk/pk conjuncts: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | other predicates: l1.l_tax < l2.l_tax | runtime filters: RF000 <- l2.l_linenumber, RF001 <- l2.l_orderkey, RF002 <- l2.l_partkey, RF003 <- l2.l_suppkey | mem-estimate=98.39GB mem-reservation=136.00MB spill-buffer=8.00MB | tuple-ids=0,1 row-size=534B cardinality=360011594 | |--04:EXCHANGE [HASH(l2.l_linenumber,l2.l_orderkey,l2.l_partkey,l2.l_suppkey)] | | mem-estimate=0B mem-reservation=0B | | tuple-ids=1 row-size=267B cardinality=360011594 | | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 | Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B | 01:SCAN HDFS [tpch_60_parquet.lineitem l2, RANDOM] | partitions=1/1 files=51 size=12.56GB | stats-rows=360011594 extrapolated-rows=disabled | table stats: rows=360011594 size=12.56GB | column stats: all | mem-estimate=704.00MB mem-reservation=0B | tuple-ids=1 row-size=267B cardinality=360011594 | 03:EXCHANGE [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] | mem-estimate=0B mem-reservation=0B | tuple-ids=0 row-size=267B cardinality=360011594 | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B 00:SCAN HDFS [tpch_60_parquet.lineitem l1, RANDOM] partitions=1/1 files=51 size=12.56GB runtime filters: RF000 -> l1.l_linenumber, RF001 -> l1.l_orderkey, RF002 -> l1.l_partkey, RF003 -> l1.l_suppkey stats-rows=360011594 extrapolated-rows=disabled table stats: rows=360011594 size=12.56GB column stats: all mem-estimate=704.00MB mem-reservation=0B tuple-ids=0 row-size=267B cardinality=360011594 ---------------- Estimated Per-Host Mem: 107126659482 Per-Host Memory Reservation: 142606336 Request Pool: default-pool Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail --------------------------------------------------------------------------------------------------------------------------- 05:EXCHANGE 1 6.997us 6.997us 0 360.01M 0 0 UNPARTITIONED 02:HASH JOIN 1 30m47s 30m47s 0 360.01M 8.00 GB 98.39 GB INNER JOIN, PARTITIONED |--04:EXCHANGE 1 7s206ms 7s206ms 360.01M 360.01M 0 0 HASH(l2.l_linenumber,l2.l_o... | 01:SCAN HDFS 1 7m19s 7m19s 360.01M 360.01M 1.11 GB 704.00 MB tpch_60_parquet.lineitem l2 03:EXCHANGE 1 7s998ms 7s998ms 360.01M 360.01M 0 0 HASH(l1.l_linenumber,l1.l_o... 00:SCAN HDFS 1 2m9s 2m9s 360.01M 360.01M 1.12 GB 704.00 MB tpch_60_parquet.lineitem l1 Errors: Planner Timeline: 20.960ms - Analysis finished: 2.827ms (2.827ms) - Equivalence classes computed: 5.251ms (2.424ms) - Single node plan created: 12.535ms (7.283ms) - Runtime filters computed: 12.894ms (359.269us) - Distributed plan created: 13.155ms (260.630us) - Planning finished: 20.960ms (7.805ms) Query Timeline: 45m24s - Query submitted: 15.821us (15.821us) - Planning finished: 23.138ms (23.122ms) - Submit for admission: 23.296ms (158.393us) - Completed admission: 23.326ms (29.775us) - Ready to start on 1 backends: 23.426ms (100.281us) - All 1 execution backends (4 fragment instances) started: 24.145ms (718.281us) - First dynamic filter received: 12m7s (12m7s) - Rows available: 45m23s (33m16s) - First row fetched: 45m24s (548.389ms) - Unregister query: 45m24s (1.850ms) - ComputeScanRangeAssignmentTimer: 56.868us ImpalaServer: - ClientFetchWaitTimer: 549.401ms - RowMaterializationTimer: 823.726us Execution Profile 7d4ad07a20dbeeed:6f4174f00000000:(Total: 45m23s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 4 Filter routing table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 1 (1) N/A N/A true 2 2 0 REMOTE false 1 (1) N/A N/A true 1 2 0 REMOTE false 1 (1) N/A N/A true 0 2 0 REMOTE false 1 (1) N/A N/A true Backend startup latencies: Count: 1, min / max: 1ms / 1ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms Final filter table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 0 (1) 12m7s 12m7s false 2 2 0 REMOTE false 0 (1) 12m7s 12m7s false 1 2 0 REMOTE false 0 (1) 12m7s 12m7s false 0 2 0 REMOTE false 0 (1) 12m7s 12m7s false Per Node Peak Memory Usage: tarmstrong-box2:22000(9.67 GB) - FiltersReceived: 4 (4) - FinalizationTimer: 0.000ns Averaged Fragment F03:(Total: 45m23s, non-child: 270.288us, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:45m23s max:45m23s mean: 45m23s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.67 GB (10381861291) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 45m23s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 45m23s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 42.105us - ExecTreeExecTime: 6.139us - OpenTime: 45m23s - ExecTreeOpenTime: 45m23s - PrepareTime: 11.217ms - ExecTreePrepareTime: 10.969us PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 11.031ms, non-child: 11.031ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 10.533ms EXCHANGE_NODE (id=5):(Total: 45m23s, non-child: 45m23s, % non-child: 100.00%) - BytesReceived: 0 - ConvertRowBatchTime: 1.415us - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 45m23s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Coordinator Fragment F03: Instance 7d4ad07a20dbeeed:6f4174f00000000 (host=tarmstrong-box2:22000):(Total: 45m23s, non-child: 270.288us, % non-child: 0.00%) MemoryUsage(1m4s): 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.67 GB (10381861291) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 45m23s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 45m23s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 42.105us - ExecTreeExecTime: 6.139us - OpenTime: 45m23s - ExecTreeOpenTime: 45m23s - PrepareTime: 11.217ms - ExecTreePrepareTime: 10.969us PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 11.031ms, non-child: 11.031ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 10.533ms EXCHANGE_NODE (id=5):(Total: 45m23s, non-child: 6.997us, % non-child: 0.00%) BytesReceived(1m4s): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 0 - ConvertRowBatchTime: 1.415us - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 45m23s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F02:(Total: 45m23s, non-child: 5s649ms, % non-child: 0.21%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:45m23s max:45m23s mean: 45m23s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 1.27 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.11 GB (8706768364) - PeakReservation: 8.00 GB (8589934592) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.67 GB (10381861291) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 14m25s - TotalNetworkSendTime: 323.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 22.67K (22674) - TotalThreadsTotalWallClockTime: 57m33s - TotalThreadsSysTime: 22s604ms - TotalThreadsUserTime: 5m10s - TotalThreadsVoluntaryContextSwitches: 684.93K (684928) Fragment Instance Lifecycle Timings: - ExecTime: 33m14s - ExecTreeExecTime: 33m8s - OpenTime: 12m9s - ExecTreeOpenTime: 12m9s - PrepareTime: 16.394ms - ExecTreePrepareTime: 60.571us DataStreamSender (dst_id=5):(Total: 6.097us, non-child: 6.097us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 172.750ms, non-child: 172.750ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 47.267ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 110 (110) - NumInstructions: 5.01K (5009) - OptimizationTime: 111.385ms - PeakMemoryUsage: 2.45 MB (2564608) - PrepareTime: 13.759ms HASH_JOIN_NODE (id=2):(Total: 45m18s, non-child: 30m47s, % non-child: 67.96%) - BuildRows: 360.01M (360011594) - BuildTime: 2m43s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.00 GB (8590510412) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 3m51s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Buffer pool: - AllocTime: 930.578ms - CumulativeAllocationBytes: 357.80 GB (384181469184) - CumulativeAllocations: 43.77K (43766) - PeakReservation: 8.00 GB (8589934592) - PeakUnpinnedBytes: 169.08 GB (181546254336) - PeakUsedReservation: 8.00 GB (8589934592) - ReadIoBytes: 167.08 GB (179398770688) - ReadIoOps: 21.39K (21386) - ReadIoWaitTime: 22m38s - WriteIoBytes: 169.08 GB (181546254336) - WriteIoOps: 21.64K (21642) - WriteIoWaitTime: 2m57s Hash Join Builder (join_node_id=2): - BuildRowsPartitionTime: 2m40s - BuildRowsPartitioned: 360.01M (360011594) - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 14m57s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 48.12 KB (49280) - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) EXCHANGE_NODE (id=4):(Total: 9m20s, non-child: 9m20s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 2s837ms - DeserializeRowBatchTimer: 1m13s - FirstBatchArrivalWaitTime: 139.108ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 642.50 K/sec - SendersBlockedTimer: 54s898ms - SendersBlockedTotalTimer(*): 54s898ms EXCHANGE_NODE (id=3):(Total: 5m11s, non-child: 5m11s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s051ms - DeserializeRowBatchTimer: 1m21s - FirstBatchArrivalWaitTime: 1s436ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.16 M/sec - SendersBlockedTimer: 13m46s - SendersBlockedTotalTimer(*): 13m46s Fragment F02: Instance 7d4ad07a20dbeeed:6f4174f00000003 (host=tarmstrong-box2:22000):(Total: 45m23s, non-child: 5s649ms, % non-child: 0.21%) MemoryUsage(1m4s): 6.81 GB, 8.10 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.08 GB, 8.07 GB, 8.07 GB, 8.07 GB, 8.08 GB, 8.07 GB, 8.08 GB, 8.07 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB ThreadUsage(1m4s): 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.27 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.11 GB (8706768364) - PeakReservation: 8.00 GB (8589934592) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.67 GB (10381861291) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 14m25s - TotalNetworkSendTime: 323.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 22.67K (22674) - TotalThreadsTotalWallClockTime: 57m33s - TotalThreadsSysTime: 22s604ms - TotalThreadsUserTime: 5m10s - TotalThreadsVoluntaryContextSwitches: 684.93K (684928) Fragment Instance Lifecycle Timings: - ExecTime: 33m14s - ExecTreeExecTime: 33m8s - OpenTime: 12m9s - ExecTreeOpenTime: 12m9s - PrepareTime: 16.394ms - ExecTreePrepareTime: 60.571us DataStreamSender (dst_id=5):(Total: 6.097us, non-child: 6.097us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 172.750ms, non-child: 172.750ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 47.267ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 110 (110) - NumInstructions: 5.01K (5009) - OptimizationTime: 111.385ms - PeakMemoryUsage: 2.45 MB (2564608) - PrepareTime: 13.759ms HASH_JOIN_NODE (id=2):(Total: 45m18s, non-child: 30m47s, % non-child: 67.96%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 360.01M (360011594) - BuildTime: 2m43s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.00 GB (8590510412) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 3m51s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Buffer pool: - AllocTime: 930.578ms - CumulativeAllocationBytes: 357.80 GB (384181469184) - CumulativeAllocations: 43.77K (43766) - PeakReservation: 8.00 GB (8589934592) - PeakUnpinnedBytes: 169.08 GB (181546254336) - PeakUsedReservation: 8.00 GB (8589934592) - ReadIoBytes: 167.08 GB (179398770688) - ReadIoOps: 21.39K (21386) - ReadIoWaitTime: 22m38s - WriteIoBytes: 169.08 GB (181546254336) - WriteIoOps: 21.64K (21642) - WriteIoWaitTime: 2m57s Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled, Spilled Runtime filters: 0 of 4 Runtime Filters Published, 4 Disabled - BuildRowsPartitionTime: 2m40s - BuildRowsPartitioned: 360.01M (360011594) - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 14m57s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 48.12 KB (49280) - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) EXCHANGE_NODE (id=4):(Total: 9m20s, non-child: 7s206ms, % non-child: 1.29%) BytesReceived(16s000ms): 1.35 GB, 3.93 GB, 5.24 GB, 6.22 GB, 7.13 GB, 8.09 GB, 9.03 GB, 10.03 GB, 11.27 GB, 12.33 GB, 13.55 GB, 14.35 GB, 14.92 GB, 15.88 GB, 16.86 GB, 17.98 GB, 18.72 GB, 19.20 GB, 19.83 GB, 20.71 GB, 21.32 GB, 22.49 GB, 23.52 GB, 24.42 GB, 25.41 GB, 26.31 GB, 26.90 GB, 27.61 GB, 28.13 GB, 28.30 GB, 28.84 GB, 29.64 GB, 30.58 GB, 31.61 GB, 32.30 GB, 32.85 GB, 33.74 GB, 34.34 GB, 35.11 GB, 35.93 GB, 36.62 GB, 37.25 GB, 37.81 GB, 38.59 GB, 39.05 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 2s837ms - DeserializeRowBatchTimer: 1m13s - FirstBatchArrivalWaitTime: 139.108ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 642.50 K/sec - SendersBlockedTimer: 54s898ms - SendersBlockedTotalTimer(*): 54s898ms EXCHANGE_NODE (id=3):(Total: 5m11s, non-child: 7s998ms, % non-child: 2.57%) BytesReceived(1m4s): 9.99 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 1.10 GB, 3.64 GB, 7.03 GB, 11.36 GB, 18.24 GB, 24.58 GB, 31.51 GB, 38.03 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s051ms - DeserializeRowBatchTimer: 1m21s - FirstBatchArrivalWaitTime: 1s436ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.16 M/sec - SendersBlockedTimer: 13m46s - SendersBlockedTotalTimer(*): 13m46s Averaged Fragment F00:(Total: 20m, non-child: 13m36s, % non-child: 67.99%) split sizes: min: 12.56 GB, max: 12.56 GB, avg: 12.56 GB, stddev: 0 completion times: min:20m max:20m mean: 20m stddev:0.000ns execution rates: min:10.71 MB/sec max:10.71 MB/sec mean:10.71 MB/sec stddev:0.20 B/sec num instances: 1 - AverageThreadTokens: 5.61 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1201678433) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.67 GB (10381861291) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 13m51s - TotalStorageWaitTime: 10m33s - TotalThreadsInvoluntaryContextSwitches: 19.18K (19178) - TotalThreadsTotalWallClockTime: 1h52m - TotalThreadsSysTime: 2s192ms - TotalThreadsUserTime: 5m19s - TotalThreadsVoluntaryContextSwitches: 352.21K (352213) Fragment Instance Lifecycle Timings:.61 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1201678433) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.67 GB (10381861291) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 13m51s - TotalStorageWaitTime: 10m33s - TotalThreadsInvoluntaryContextSwitches: 19.18K (19178) - TotalThreadsTotalWallClockTime: 1h52m - TotalThreadsSysTime: 2s192ms - TotalThreadsUserTime: 5m19s - TotalThreadsVoluntaryContextSwitches: 352.21K (352213) Fragment Instance Lifecycle Timings: - ExecTime: 20m - ExecTreeExecTime: 2m9s - OpenTime: 42.246ms - ExecTreeOpenTime: 22.919us - PrepareTime: 12.145ms - ExecTreePrepareTime: 96.225us DataStreamSender (dst_id=3):(Total: 4m14s, non-child: 4m14s, % non-child: 100.00%) - BytesSent: 39.92 GB (42858904369) - NetworkThroughput(*): 2.37 GB/sec - OverallThroughput: 160.56 MB/sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 3m57s - TransmitDataRPCTime: 16s831ms - UncompressedRowBatchSize: 91.46 GB (98207171210) CodeGen:(Total: 53.487ms, non-child: 53.487ms, % non-child: 100.00%) - CodegenTime: 964.567us - CompileTime: 7.039ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 36 (36) - NumInstructions: 655 (655) - OptimizationTime: 34.524ms - PeakMemoryUsage: 327.50 KB (335360) - PrepareTime: 11.638ms HDFS_SCAN_NODE (id=0):(Total: 2m9s, non-child: 2m9s, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 51 out of 51 Runtime filters: Not all filters arrived (arrived: [], missing [0, 1, 2, 3]), waited for 1s023ms Hdfs Read Thread Concurrency Bucket: 0:76.04% 1:1.708% 2:0.9167% 3:0.7083% 4:13.75% 5:1.167% 6:5.708% File Formats: PARQUET/SNAPPY:816 BytesRead(32s000ms): 936.24 MB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.06 GB, 1.15 GB, 1.56 GB, 2.16 GB, 2.69 GB, 3.28 GB, 4.45 GB, 5.62 GB, 6.66 GB, 7.66 GB, 8.45 GB, 9.76 GB, 10.80 GB, 11.90 GB - FooterProcessingTime: (Avg: 1s446ms ; Min: 1.388ms ; Max: 8s388ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 1.09 - AverageScannerThreadConcurrency: 4.61 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s459ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) 8904369) - NetworkThroughput(*): 2.84 GB/sec - OverallThroughput: 167.33 MB/sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 3m50s - TransmitDataRPCTime: 14s044ms - UncompressedRowBatchSize: 91.46 GB (98207171210) CodeGen:(Total: 18.750ms, non-child: 18.750ms, % non-child: 100.00%) - CodegenTime: 294.454us - CompileTime: 3.124ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 6 (6) - NumInstructions: 98 (98) - OptimizationTime: 3.974ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 11.293ms HDFS_SCAN_NODE (id=1):(Total: 7m19s, non-child: 7m19s, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 1.85 - AverageScannerThreadConcurrency: 1.12 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s240ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.11 GB (1197163143) - PerReadThreadRawHdfsThroughput: 9.55 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 7m19s - RowBatchQueuePutWaitTime: 5m3s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 818.72 K/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 8.90K (8899) - ScannerThreadsTotalWallClockTime: 13m37s - MaterializeTupleTime(*): 46s448ms - ScannerThreadsSysTime: 1s616ms - ScannerThreadsUserTime: 1m13s - ScannerThreadsVoluntaryContextSwitches: 342.41K (342406) - TotalRawHdfsReadTime(*): 22m26s - TotalReadThroughput: 17.69 MB/sec Fragment F01: Instance 7d4ad07a20dbeeed:6f4174f00000002 (host=tarmstrong-box2:22000):(Total: 12m7s, non-child: 43s208ms, % non-child: 5.94%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB MemoryUsage(16s000ms): 889.22 MB, 301.20 MB, 163.70 MB, 180.10 MB, 159.65 MB, 174.35 MB, 125.25 MB, 153.58 MB, 113.09 MB, 131.57 MB, 151.52 MB, 163.73 MB, 90.86 MB, 138.72 MB, 166.40 MB, 138.38 MB, 104.54 MB, 130.97 MB, 154.93 MB, 128.10 MB, 119.31 MB, 151.96 MB, 123.00 MB, 65.95 MB, 151.63 MB, 111.61 MB, 97.15 MB, 96.93 MB, 67.65 MB, 184.38 MB, 127.51 MB, 67.65 MB, 155.94 MB, 163.78 MB, 178.99 MB, 64.95 MB, 146.41 MB, 161.59 MB, 115.73 MB, 58.66 MB, 87.80 MB, 98.52 MB, 111.26 MB, 159.16 MB, 84.78 MB ThreadUsage(16s000ms): 6, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 - AverageThreadTokens: 2.12 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1197398330) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.67 GB (10381861291) - RowsProduced: 360.01M (360011594) -child: 7m19s, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 51 out of 51 Hdfs Read Thread Concurrency Bucket: 0:38.24% 1:35.49% 2:5.915% 3:0.5502% 4:5.158% 5:0.2751% 6:14.37% File Formats: PARQUET/SNAPPY:816 BytesRead(16s000ms): 1.01 GB, 1.36 GB, 1.74 GB, 2.06 GB, 2.33 GB, 2.64 GB, 2.91 GB, 3.24 GB, 3.61 GB, 3.95 GB, 4.33 GB, 4.60 GB, 4.73 GB, 5.08 GB, 5.39 GB, 5.72 GB, 5.94 GB, 6.10 GB, 6.31 GB, 6.58 GB, 6.77 GB, 7.15 GB, 7.45 GB, 7.71 GB, 8.08 GB, 8.33 GB, 8.50 GB, 8.73 GB, 8.87 GB, 9.01 GB, 9.14 GB, 9.35 GB, 9.70 GB, 10.03 GB, 10.27 GB, 10.36 GB, 10.69 GB, 10.89 GB, 11.10 GB, 11.32 GB, 11.56 GB, 11.77 GB, 11.94 GB, 12.23 GB, 12.33 GB - FooterProcessingTime: (Avg: 1s430ms ; Min: 1.069ms ; Max: 4s557ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 1.85 - AverageScannerThreadConcurrency: 1.12 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s240ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.11 GB (1197163143) - PerReadThreadRawHdfsThroughput: 9.55 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 7m19s - RowBatchQueuePutWaitTime: 5m3s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 818.72 K/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 8.90K (8899) - ScannerThreadsTotalWallClockTime: 13m37s - MaterializeTupleTime(*): 46s448ms - ScannerThreadsSysTime: 1s616ms - ScannerThreadsUserTime: 1m13s - ScannerThreadsVoluntaryContextSwitches: 342.41K (342406) - TotalRawHdfsReadTime(*): 22m26s - TotalReadThroughput: 17.69 MB/sec