Query (id=cb406cc0cb0413a0:c9ed373b00000000): Summary: Session ID: 2e472a715692f619:dd5dd64066c604aa Session Type: BEESWAX Start Time: 2017-08-17 21:40:02.620427000 End Time: 2017-08-17 22:20:28.816577000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.10.0-SNAPSHOT RELEASE (build d5b0c6b9366bc54801ac0dc74c942cdbda213b67) User: tarmstrong Connected User: tarmstrong Delegated User: Network Address: ::ffff:127.0.0.1:60092 Default Db: tpch_60_parquet Sql Statement: select * from lineitem l1 join lineitem l2 on l1.l_orderkey = l2.l_orderkey and l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey and l1.l_linenumber = l2.l_linenumber where l1.l_tax < l2.l_tax Coordinator: tarmstrong-box2:22000 Query Options (non default): MEM_LIMIT=10737418240,MT_DOP=0 Plan: ---------------- Per-Host Resource Reservation: Memory=136.00MB Per-Host Resource Estimates: Memory=99.77GB F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=0B mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 05:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=0,1 row-size=534B cardinality=360011594 | F02:PLAN FRAGMENT [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] hosts=1 instances=1 Per-Host Resources: mem-estimate=98.39GB mem-reservation=136.00MB 02:HASH JOIN [INNER JOIN, PARTITIONED] | hash predicates: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | fk/pk conjuncts: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | other predicates: l1.l_tax < l2.l_tax | runtime filters: RF000 <- l2.l_linenumber, RF001 <- l2.l_orderkey, RF002 <- l2.l_partkey, RF003 <- l2.l_suppkey | mem-estimate=98.39GB mem-reservation=136.00MB | tuple-ids=0,1 row-size=534B cardinality=360011594 | |--04:EXCHANGE [HASH(l2.l_linenumber,l2.l_orderkey,l2.l_partkey,l2.l_suppkey)] | | mem-estimate=0B mem-reservation=0B | | tuple-ids=1 row-size=267B cardinality=360011594 | | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 | Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B | 01:SCAN HDFS [tpch_60_parquet.lineitem l2, RANDOM] | partitions=1/1 files=51 size=12.56GB | stats-rows=360011594 extrapolated-rows=disabled | table stats: rows=360011594 size=12.56GB | column stats: all | mem-estimate=704.00MB mem-reservation=0B | tuple-ids=1 row-size=267B cardinality=360011594 | 03:EXCHANGE [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] | mem-estimate=0B mem-reservation=0B | tuple-ids=0 row-size=267B cardinality=360011594 | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B 00:SCAN HDFS [tpch_60_parquet.lineitem l1, RANDOM] partitions=1/1 files=51 size=12.56GB runtime filters: RF000 -> l1.l_linenumber, RF001 -> l1.l_orderkey, RF002 -> l1.l_partkey, RF003 -> l1.l_suppkey stats-rows=360011594 extrapolated-rows=disabled table stats: rows=360011594 size=12.56GB column stats: all mem-estimate=704.00MB mem-reservation=0B tuple-ids=0 row-size=267B cardinality=360011594 ---------------- Estimated Per-Host Mem: 107126659482 Per-Host Memory Reservation: 142606336 Request Pool: default-pool Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail --------------------------------------------------------------------------------------------------------------------------- 05:EXCHANGE 1 3.656us 3.656us 0 360.01M 0 0 UNPARTITIONED 02:HASH JOIN 1 32m53s 32m53s 0 360.01M 8.01 GB 98.39 GB INNER JOIN, PARTITIONED |--04:EXCHANGE 1 8s894ms 8s894ms 360.01M 360.01M 0 0 HASH(l2.l_linenumber,l2.l_o... | 01:SCAN HDFS 1 2m2s 2m2s 360.01M 360.01M 1.12 GB 704.00 MB tpch_60_parquet.lineitem l2 03:EXCHANGE 1 10s228ms 10s228ms 360.01M 360.01M 0 0 HASH(l1.l_linenumber,l1.l_o... 00:SCAN HDFS 1 39s680ms 39s680ms 360.01M 360.01M 1.12 GB 704.00 MB tpch_60_parquet.lineitem l1 Errors: Planner Timeline: 25.002ms - Analysis finished: 4.300ms (4.300ms) - Equivalence classes computed: 7.694ms (3.393ms) - Single node plan created: 16.358ms (8.663ms) - Runtime filters computed: 16.719ms (361.766us) - Distributed plan created: 16.990ms (270.743us) - Planning finished: 25.002ms (8.011ms) Query Timeline: 40m26s - Query submitted: 28.672us (28.672us) - Planning finished: 27.654ms (27.626ms) - Submit for admission: 27.839ms (184.443us) - Completed admission: 27.885ms (46.386us) - Ready to start on 1 backends: 28.004ms (118.758us) - All 1 execution backends (4 fragment instances) started: 28.650ms (646.195us) - First dynamic filter received: 8m21s (8m21s) - Rows available: 40m25s (32m3s) - First row fetched: 40m26s (980.480ms) - Unregister query: 40m26s (3.586ms) - ComputeScanRangeAssignmentTimer: 59.246us ImpalaServer: - ClientFetchWaitTimer: 982.389ms - RowMaterializationTimer: 1.636ms Execution Profile cb406cc0cb0413a0:c9ed373b00000000:(Total: 40m25s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 4 Filter routing table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 1 (1) N/A N/A true 2 2 0 REMOTE false 1 (1) N/A N/A true 1 2 0 REMOTE false 1 (1) N/A N/A true 0 2 0 REMOTE false 1 (1) N/A N/A true Backend startup latencies: Count: 1, min / max: 1ms / 1ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms Final filter table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 0 (1) 8m21s 8m21s false 2 2 0 REMOTE false 0 (1) 8m21s 8m21s false 1 2 0 REMOTE false 0 (1) 8m21s 8m21s false 0 2 0 REMOTE false 0 (1) 8m21s 8m21s false Per Node Peak Memory Usage: tarmstrong-box2:22000(9.68 GB) - FiltersReceived: 4 (4) - FinalizationTimer: 0.000ns Averaged Fragment F03:(Total: 40m25s, non-child: 238.459us, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:40m25s max:40m25s mean: 40m25s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PerHostPeakMemUsage: 9.68 GB (10390080023) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 40m25s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 40m25s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 5.731us - ExecTreeExecTime: 2.439us - OpenTime: 40m25s - ExecTreeOpenTime: 40m25s - PrepareTime: 11.456ms - ExecTreePrepareTime: 11.841us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 22.39K (22395) - BlocksRecycled: 15 (15) - BufferedPins: 170 (170) - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 8.00 GB (8589934592) - PeakMemoryUsage: 8.00 GB (8589934592) - ScratchBytesRead: 167.74 GB (180111802368) - ScratchBytesWritten: 167.74 GB (180111802368) - ScratchFileUsedBytes: 167.74 GB (180111802368) - ScratchReads: 21.47K (21471) - ScratchWrites: 21.47K (21471) - TotalBufferWaitTime: 7m30s - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 20m32s PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 11.267ms, non-child: 11.267ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 10.061ms EXCHANGE_NODE (id=5):(Total: 40m25s, non-child: 40m25s, % non-child: 100.00%) - BytesReceived: 0 - ConvertRowBatchTime: 675.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 40m25s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Coordinator Fragment F03: Instance cb406cc0cb0413a0:c9ed373b00000000 (host=tarmstrong-box2:22000):(Total: 40m25s, non-child: 238.459us, % non-child: 0.00%) MemoryUsage(1m4s): 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PerHostPeakMemUsage: 9.68 GB (10390080023) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 40m25s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 40m25s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 5.731us - ExecTreeExecTime: 2.439us - OpenTime: 40m25s - ExecTreeOpenTime: 40m25s - PrepareTime: 11.456ms - ExecTreePrepareTime: 11.841us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 22.39K (22395) - BlocksRecycled: 15 (15) - BufferedPins: 170 (170) - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 8.00 GB (8589934592) - PeakMemoryUsage: 8.00 GB (8589934592) - ScratchBytesRead: 167.74 GB (180111802368) - ScratchBytesWritten: 167.74 GB (180111802368) - ScratchFileUsedBytes: 167.74 GB (180111802368) - ScratchReads: 21.47K (21471) - ScratchWrites: 21.47K (21471) - TotalBufferWaitTime: 7m30s - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 20m32s PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 11.267ms, non-child: 11.267ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 10.061ms EXCHANGE_NODE (id=5):(Total: 40m25s, non-child: 3.656us, % non-child: 0.00%) BytesReceived(1m4s): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 0 - ConvertRowBatchTime: 675.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 40m25s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F02:(Total: 40m25s, non-child: 5s186ms, % non-child: 0.21%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:40m25s max:40m25s mean: 40m25s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 1.21 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.12 GB (8715916648) - PerHostPeakMemUsage: 9.68 GB (10390080023) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 7m20s - TotalNetworkSendTime: 127.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 22.61K (22608) - TotalThreadsTotalWallClockTime: 48m53s - TotalThreadsSysTime: 16s248ms - TotalThreadsUserTime: 4m54s - TotalThreadsVoluntaryContextSwitches: 654.19K (654190) Fragment Instance Lifecycle Timings: - ExecTime: 31m56s - ExecTreeExecTime: 31m51s - OpenTime: 8m29s - ExecTreeOpenTime: 8m28s - PrepareTime: 24.297ms - ExecTreePrepareTime: 96.989us DataStreamSender (dst_id=5):(Total: 6.495us, non-child: 6.495us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 179.556ms, non-child: 179.556ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 48.182ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 112 (112) - NumInstructions: 4.99K (4991) - OptimizationTime: 110.665ms - PeakMemoryUsage: 2.44 MB (2555392) - PrepareTime: 20.296ms HASH_JOIN_NODE (id=2):(Total: 40m19s, non-child: 32m53s, % non-child: 81.55%) - BuildRows: 360.01M (360011594) - BuildTime: 4m20s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.01 GB (8599746911) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 6m42s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Hash Join Builder (join_node_id=2): - BuildRowsPartitionTime: 4m13s - BuildRowsPartitioned: 360.01M (360011594) - GetNewBlockTime: 7m25s - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 10m40s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 8.01 GB (8599412864) - PinTime: 20m33s - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) - UnpinTime: 6.886ms EXCHANGE_NODE (id=4):(Total: 4m1s, non-child: 4m1s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s035ms - DeserializeRowBatchTimer: 1m17s - FirstBatchArrivalWaitTime: 118.971ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.49 M/sec - SendersBlockedTimer: 2m19s - SendersBlockedTotalTimer(*): 2m19s EXCHANGE_NODE (id=3):(Total: 3m26s, non-child: 3m26s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s388ms - DeserializeRowBatchTimer: 1m29s - FirstBatchArrivalWaitTime: 1s408ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.75 M/sec - SendersBlockedTimer: 12m28s - SendersBlockedTotalTimer(*): 12m28s Fragment F02: Instance cb406cc0cb0413a0:c9ed373b00000003 (host=tarmstrong-box2:22000):(Total: 40m25s, non-child: 5s186ms, % non-child: 0.21%) MemoryUsage(1m4s): 6.50 GB, 7.36 GB, 7.16 GB, 6.27 GB, 7.09 GB, 4.57 GB, 5.19 GB, 5.73 GB, 6.84 GB, 6.84 GB, 6.84 GB, 6.85 GB, 6.85 GB, 6.85 GB, 6.85 GB, 6.85 GB, 4.12 GB, 5.47 GB, 4.91 GB, 4.17 GB, 5.86 GB, 5.13 GB, 4.56 GB, 5.08 GB, 6.30 GB, 4.79 GB, 4.31 GB, 5.19 GB, 5.57 GB, 4.34 GB, 5.04 GB, 5.98 GB, 3.99 GB, 4.99 GB, 6.18 GB, 4.59 GB, 4.38 GB ThreadUsage(1m4s): 1, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.21 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.12 GB (8715916648) - PerHostPeakMemUsage: 9.68 GB (10390080023) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 7m20s - TotalNetworkSendTime: 127.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 22.61K (22608) - TotalThreadsTotalWallClockTime: 48m53s - TotalThreadsSysTime: 16s248ms - TotalThreadsUserTime: 4m54s - TotalThreadsVoluntaryContextSwitches: 654.19K (654190) Fragment Instance Lifecycle Timings: - ExecTime: 31m56s - ExecTreeExecTime: 31m51s - OpenTime: 8m29s - ExecTreeOpenTime: 8m28s - PrepareTime: 24.297ms - ExecTreePrepareTime: 96.989us DataStreamSender (dst_id=5):(Total: 6.495us, non-child: 6.495us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 179.556ms, non-child: 179.556ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 48.182ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 112 (112) - NumInstructions: 4.99K (4991) - OptimizationTime: 110.665ms - PeakMemoryUsage: 2.44 MB (2555392) - PrepareTime: 20.296ms HASH_JOIN_NODE (id=2):(Total: 40m19s, non-child: 32m53s, % non-child: 81.55%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 360.01M (360011594) - BuildTime: 4m20s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.01 GB (8599746911) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 6m42s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled, Spilled Runtime filters: 0 of 4 Runtime Filters Published, 4 Disabled - BuildRowsPartitionTime: 4m13s - BuildRowsPartitioned: 360.01M (360011594) - GetNewBlockTime: 7m25s - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 10m40s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 8.01 GB (8599412864) - PinTime: 20m33s - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) - UnpinTime: 6.886ms EXCHANGE_NODE (id=4):(Total: 4m1s, non-child: 8s894ms, % non-child: 3.69%) BytesReceived(8s000ms): 577.43 MB, 1.97 GB, 3.37 GB, 4.46 GB, 5.31 GB, 6.12 GB, 6.73 GB, 7.55 GB, 8.24 GB, 8.94 GB, 9.54 GB, 10.17 GB, 11.00 GB, 11.73 GB, 12.41 GB, 13.01 GB, 13.69 GB, 14.48 GB, 15.15 GB, 15.82 GB, 16.50 GB, 17.19 GB, 17.84 GB, 18.54 GB, 19.28 GB, 19.92 GB, 20.55 GB, 21.17 GB, 21.81 GB, 22.36 GB, 22.91 GB, 23.53 GB, 24.18 GB, 24.72 GB, 25.36 GB, 25.89 GB, 26.57 GB, 27.19 GB, 27.63 GB, 28.24 GB, 28.81 GB, 29.41 GB, 29.86 GB, 30.52 GB, 31.11 GB, 31.70 GB, 32.36 GB, 32.95 GB, 33.37 GB, 34.05 GB, 34.46 GB, 35.16 GB, 35.56 GB, 35.95 GB, 36.41 GB, 36.79 GB, 37.13 GB, 37.32 GB, 37.93 GB, 38.19 GB, 38.92 GB, 39.17 GB, 39.73 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s035ms - DeserializeRowBatchTimer: 1m17s - FirstBatchArrivalWaitTime: 118.971ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.49 M/sec - SendersBlockedTimer: 2m19s - SendersBlockedTotalTimer(*): 2m19s EXCHANGE_NODE (id=3):(Total: 3m26s, non-child: 10s228ms, % non-child: 4.96%) BytesReceived(1m4s): 9.91 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 15.59 MB, 2.26 GB, 5.91 GB, 9.79 GB, 14.38 GB, 19.98 GB, 25.24 GB, 30.49 GB, 35.64 GB, 39.61 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s388ms - DeserializeRowBatchTimer: 1m29s - FirstBatchArrivalWaitTime: 1s408ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.75 M/sec - SendersBlockedTimer: 12m28s - SendersBlockedTotalTimer(*): 12m28s Averaged Fragment F00:(Total: 17m26s, non-child: 12m24s, % non-child: 71.18%) split sizes: min: 12.56 GB, max: 12.56 GB, avg: 12.56 GB, stddev: 0 completion times: min:17m26s max:17m26s mean: 17m26s stddev:0.000ns execution rates: min:12.29 MB/sec max:12.29 MB/sec mean:12.29 MB/sec stddev:0.18 B/sec num instances: 1 - AverageThreadTokens: 5.51 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1197479377) - PerHostPeakMemUsage: 9.68 GB (10390080023) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 12m39s - TotalStorageWaitTime: 4m57s - TotalThreadsInvoluntaryContextSwitches: 21.91K (21905) - TotalThreadsTotalWallClockTime: 1h36m - TotalThreadsSysTime: 2s248ms - TotalThreadsUserTime: 5m31s - TotalThreadsVoluntaryContextSwitches: 354.07K (354069) Fragment Instance Lifecycle Timings: - ExecTime: 17m2.343ms - ExecTreeOpenTime: 26.522us - PrepareTime: 12.101ms - ExecTreePrepareTime: 115.611us DataStreamSender (dst_id=3):(Total: 4m21s, non-child: 4m21s, % non-child: 100.00%) - BytesSent: 39.92 GB (42858904369) - NetworkThroughput(*): 2.48 GB/sec - OverallThroughput: 156.11 MB/sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 4m5s - TransmitDataRPCTime: 16s108ms - UncompressedRowBatchSize: 91.46 GB (98207171210) CodeGen:(Total: 63.619ms, non-child: 63.619ms, % non-child: 100.00%) - CodegenTime: 1.004ms - CompileTime: 11.698ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 36 (36) - NumInstructions: 655 (655) - OptimizationTime: 40.179ms - PeakMemoryUsage: 327.50 KB (335360) - PrepareTime: 11.442ms HDFS_SCAN_NODE (id=0):(Total: 39s680ms, non-child: 39s680ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 51 out of 51 Runtime filters: Not all filters arrived (arrived: [], missing [0, 1, 2, 3]), waited for 1s003ms Hdfs Read Thread Concurrency Bucket: 0:79.25% 1:2.438% 2:1.291% 3:1.482% 4:5.497% 5:3.728% 6:6.31% File Formats: PARQUET/SNAPPY:816 BytesRead(32s000ms): 926.99 MB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.00 GB, 1.02 GB, 1.22 GB, 1.92 GB, 2.42 GB, 3.11 GB, 3.66 GB, 4.38 GB, 5.26 GB, 6.24 GB, 7.09 GB, 7.98 GB, 8.74 GB, 9.59 GB, 10.37 GB, 11.19 GB, 11.99 GB - FooterProcessingTime: (Avg: 1s333ms ; Min: 1.448ms ; Max: 10s336ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 0.96 - AverageScannerThreadConcurrency: 4.51 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 20s047ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.12 GB (1197345972) - PerReadThreadRawHdfsThroughput: 12.67 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 37s935ms - RowBatchQueuePutWaitTime: 1h12m - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 9.07 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 15.31K (15311) - ScannerThreadsTotalWallClockTime: 1h18m - MaterializeTupleTime(*): 57s076ms - ScannerThreadsSysTime: 1s228ms - ScannerThreadsUserTime: 1m24s - ScannerThreadsVoluntaryContextSwitches: 351.74K (351744) - TotalRawHdfsReadTime(*): 16m55s - TotalRe - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 6 (6) - NumInstructions: 98 (98) - OptimizationTime: 4.157ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 10.731ms HDFS_SCAN_NODE (id=1):(Total: 2m2s, non-child: 2m2s, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.67 - AverageScannerThreadConcurrency: 1.18 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s550ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.12 GB (1200471492) - PerReadThreadRawHdfsThroughput: 37.78 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 2m1s - RowBatchQueuePutWaitTime: 6m32s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.94 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 13.23K (13235) - ScannerThreadsTotalWallClockTime: 9m51s - MaterializeTupleTime(*): 46s909ms - ScannerThreadsSysTime: 1s400ms - ScannerThreadsUserTime: 1m14s - ScannerThreadsVoluntaryContextSwitches: 345.47K (345473) - TotalRawHdfsReadTime(*): 5m40s - TotalReadThroughput: 25.62 MB/sec Fragment F01: Instance cb406cc0cb0413a0:c9ed373b00000002 (host=tarmstrong-box2:22000):(Total: 8m21s, non-child: 2m8s, % non-child: 25.63%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB MemoryUsage(8s000ms): 880.23 MB, 868.67 MB, 478.16 MB, 168.33 MB, 164.33 MB, 152.78 MB, 179.71 MB, 166.57 MB, 167.64 MB, 178.98 MB, 160.05 MB, 174.63 MB, 161.64 MB, 181.40 MB, 165.39 MB, 167.34 MB, 163.61 MB, 178.44 MB, 180.75 MB, 160.82 MB, 157.95 MB, 168.93 MB, 177.16 MB, 166.56 MB, 169.39 MB, 199.32 MB, 183.62 MB, 175.46 MB, 165.11 MB, 192.91 MB, 168.52 MB, 151.78 MB, 103.28 MB, 164.13 MB, 131.73 MB, 162.28 MB, 156.70 MB, 171.10 MB, 80.20 MB, 140.18 MB, 129.15 MB, 149.17 MB, 179.95 MB, 137.82 MB, 139.89 MB, 149.40 MB, 144.25 MB, 107.05 MB, 134.71 MB, 161.04 MB, 127.42 MB, 111.04 MB, 163.77 MB, 124.40 MB, 156.32 MB, 187.13 MB, 162.63 MB, 107.37 MB, 176.21 MB, 162.16 MB, 99.75 MB, 88.23 MB ThreadUsage(8s000ms): 5, 6, 5, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 - AverageThreadTokens: 2.18 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1200679262) - PerHostPeakMemUsage: 9.68 GB (10390080023) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 2m21s - TotalStorageWaitTime: 2m9s - TotalThreadsInvoluntaryContextSwitches: 19.62K (19623) - TotalThreadsTotalWallClockTime: 18m13s - TotalThreadsSysTime: 2s116ms - TotalThreadsUserTime: 5m10s B, 5.92 GB, 6.16 GB, 6.40 GB, 6.58 GB, 6.77 GB, 6.95 GB, 7.15 GB, 7.31 GB, 7.49 GB, 7.65 GB, 7.87 GB, 8.05 GB, 8.24 GB, 8.44 GB, 8.64 GB, 8.74 GB, 8.97 GB, 9.14 GB, 9.32 GB, 9.51 GB, 9.67 GB, 9.87 GB, 10.06 GB, 10.26 GB, 10.44 GB, 10.56 GB, 10.80 GB, 10.90 GB, 11.12 GB, 11.28 GB, 11.39 GB, 11.53 GB, 11.69 GB, 11.77 GB, 11.80 GB, 12.03 GB, 12.10 GB, 12.30 GB, 12.37 GB - FooterProcessingTime: (Avg: 629.013ms ; Min: 2.254ms ; Max: 3s699ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 0.67 - AverageScannerThreadConcurrency: 1.18 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s550ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.12 GB (1200471492) - PerReadThreadRawHdfsThroughput: 37.78 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 2m1s - RowBatchQueuePutWaitTime: 6m32s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.94 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 13.23K (13235) - ScannerThreadsTotalWallClockTime: 9m51s - MaterializeTupleTime(*): 46s909ms - ScannerThreadsSysTime: 1s400ms - ScannerThreadsUserTime: 1m14s - ScannerThreadsVoluntaryContextSwitches: 345.47K (345473) - TotalRawHdfsReadTime(*): 5m40s - TotalReadThroughput: 25.62 MB/sec Query (id=a540d26d367b2564:c64481bc00000000): Summary: Session ID: 2e472a715692f619:dd5dd64066c604aa Session Type: BEESWAX Start Time: 2017-08-17 21:04:33.801716000 End Time: 2017-08-17 21:40:02.578511000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.10.0-SNAPSHOT RELEASE (build d5b0c6b9366bc54801ac0dc74c942cdbda213b67) User: tarmstrong Connected User: tarmstrong Delegated User: Network Address: ::ffff:127.0.0.1:60092 Default Db: tpch_60_parquet Sql Statement: select * from lineitem l1 join lineitem l2 on l1.l_orderkey = l2.l_orderkey and l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey and l1.l_linenumber = l2.l_linenumber where l1.l_tax < l2.l_tax Coordinator: tarmstrong-box2:22000 Query Options (non default): MEM_LIMIT=10737418240,MT_DOP=0 Plan: ---------------- Per-Host Resource Reservation: Memory=136.00MB Per-Host Resource Estimates: Memory=99.77GB F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=0B mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 05:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=0,1 row-size=534B cardinality=360011594 | F02:PLAN FRAGMENT [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] hosts=1 instances=1 Per-Host Resources: mem-estimate=98.39GB mem-reservation=136.00MB 02:HASH JOIN [INNER JOIN, PARTITIONED] | hash predicates: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | fk/pk conjuncts: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | other predicates: l1.l_tax < l2.l_tax | runtime filters: RF000 <- l2.l_linenumber, RF001 <- l2.l_orderkey, RF002 <- l2.l_partkey, RF003 <- l2.l_suppkey | mem-estimate=98.39GB mem-reservation=136.00MB | tuple-ids=0,1 row-size=534B cardinality=360011594 | |--04:EXCHANGE [HASH(l2.l_linenumber,l2.l_orderkey,l2.l_partkey,l2.l_suppkey)] | | mem-estimate=0B mem-reservation=0B | | tuple-ids=1 row-size=267B cardinality=360011594 | | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 | Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B | 01:SCAN HDFS [tpch_60_parquet.lineitem l2, RANDOM] | partitions=1/1 files=51 size=12.56GB | stats-rows=360011594 extrapolated-rows=disabled | table stats: rows=360011594 size=12.56GB | column stats: all | mem-estimate=704.00MB mem-reservation=0B | tuple-ids=1 row-size=267B cardinality=360011594 | 03:EXCHANGE [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] | mem-estimate=0B mem-reservation=0B | tuple-ids=0 row-size=267B cardinality=360011594 | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B 00:SCAN HDFS [tpch_60_parquet.lineitem l1, RANDOM] partitions=1/1 files=51 size=12.56GB runtime filters: RF000 -> l1.l_linenumber, RF001 -> l1.l_orderkey, RF002 -> l1.l_partkey, RF003 -> l1.l_suppkey stats-rows=360011594 extrapolated-rows=disabled table stats: rows=360011594 size=12.56GB column stats: all mem-estimate=704.00MB mem-reservation=0B tuple-ids=0 row-size=267B cardinality=360011594 ---------------- Estimated Per-Host Mem: 107126659482 Per-Host Memory Reservation: 142606336 Request Pool: default-pool Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail --------------------------------------------------------------------------------------------------------------------------- 05:EXCHANGE 1 3.300us 3.300us 0 360.01M 0 0 UNPARTITIONED 02:HASH JOIN 1 30m9s 30m9s 0 360.01M 8.01 GB 98.39 GB INNER JOIN, PARTITIONED |--04:EXCHANGE 1 9s440ms 9s440ms 360.01M 360.01M 0 0 HASH(l2.l_linenumber,l2.l_o... | 01:SCAN HDFS 1 2s271ms 2s271ms 360.01M 360.01M 1.12 GB 704.00 MB tpch_60_parquet.lineitem l2 03:EXCHANGE 1 7s821ms 7s821ms 360.01M 360.01M 0 0 HASH(l1.l_linenumber,l1.l_o... 00:SCAN HDFS 1 12s821ms 12s821ms 360.01M 360.01M 1.07 GB 704.00 MB tpch_60_parquet.lineitem l1 Errors: Planner Timeline: 26.760ms - Analysis finished: 3.655ms (3.655ms) - Equivalence classes computed: 6.731ms (3.075ms) - Single node plan created: 15.505ms (8.774ms) - Runtime filters computed: 15.907ms (402.063us) - Distributed plan created: 16.193ms (286.409us) - Planning finished: 26.760ms (10.567ms) Query Timeline: 35m28s - Query submitted: 21.511us (21.511us) - Planning finished: 29.662ms (29.641ms) - Submit for admission: 29.827ms (164.580us) - Completed admission: 29.869ms (42.348us) - Ready to start on 1 backends: 30.001ms (131.240us) - All 1 execution backends (4 fragment instances) started: 30.673ms (671.954us) - First dynamic filter received: 6m46s (6m46s) - Rows available: 35m28s (28m41s) - First row fetched: 35m28s (140.480ms) - Unregister query: 35m28s (3.280ms) - ComputeScanRangeAssignmentTimer: 64.715us ImpalaServer: - ClientFetchWaitTimer: 142.326ms - RowMaterializationTimer: 1.396ms Execution Profile a540d26d367b2564:c64481bc00000000:(Total: 35m28s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 4 Filter routing table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 1 (1) N/A N/A true 2 2 0 REMOTE false 1 (1) N/A N/A true 1 2 0 REMOTE false 1 (1) N/A N/A true 0 2 0 REMOTE false 1 (1) N/A N/A true Backend startup latencies: Count: 1, min / max: 0 / 0, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 0, 99.9th %-ile: 0 Final filter table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 0 (1) 6m46s 6m46s false 2 2 0 REMOTE false 0 (1) 6m46s 6m46s false 1 2 0 REMOTE false 0 (1) 6m46s 6m46s false 0 2 0 REMOTE false 0 (1) 6m46s 6m46s false Per Node Peak Memory Usage: tarmstrong-box2:22000(9.65 GB) - FiltersReceived: 4 (4) - FinalizationTimer: 0.000ns Averaged Fragment F03:(Total: 35m28s, non-child: 421.708us, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:35m28s max:35m28s mean: 35m28s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PerHostPeakMemUsage: 9.65 GB (10362634827) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 35m28s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 35m28s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 5.530us - ExecTreeExecTime: 2.379us - OpenTime: 35m28s - ExecTreeOpenTime: 35m28s - PrepareTime: 17.238ms - ExecTreePrepareTime: 12.088us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 22.39K (22395) - BlocksRecycled: 15 (15) - BufferedPins: 170 (170) - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 8.00 GB (8589934592) - PeakMemoryUsage: 8.00 GB (8589934592) - ScratchBytesRead: 167.74 GB (180111802368) - ScratchBytesWritten: 167.74 GB (180111802368) - ScratchFileUsedBytes: 167.74 GB (180111802368) - ScratchReads: 21.47K (21471) - ScratchWrites: 21.47K (21471) - TotalBufferWaitTime: 6m43s - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 18m39s PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 16.864ms, non-child: 16.864ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 16.467ms EXCHANGE_NODE (id=5):(Total: 35m28s, non-child: 35m28s, % non-child: 100.00%) - BytesReceived: 0 - ConvertRowBatchTime: 926.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 35m28s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Coordinator Fragment F03: Instance a540d26d367b2564:c64481bc00000000 (host=tarmstrong-box2:22000):(Total: 35m28s, non-child: 421.708us, % non-child: 0.00%) MemoryUsage(1m4s): 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PerHostPeakMemUsage: 9.65 GB (10362634827) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 35m28s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 35m28s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 5.530us - ExecTreeExecTime: 2.379us - OpenTime: 35m28s - ExecTreeOpenTime: 35m28s - PrepareTime: 17.238ms - ExecTreePrepareTime: 12.088us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 22.39K (22395) - BlocksRecycled: 15 (15) - BufferedPins: 170 (170) - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 8.00 GB (8589934592) - PeakMemoryUsage: 8.00 GB (8589934592) - ScratchBytesRead: 167.74 GB (180111802368) - ScratchBytesWritten: 167.74 GB (180111802368) - ScratchFileUsedBytes: 167.74 GB (180111802368) - ScratchReads: 21.47K (21471) - ScratchWrites: 21.47K (21471) - TotalBufferWaitTime: 6m43s - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 18m39s PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 16.864ms, non-child: 16.864ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 16.467ms EXCHANGE_NODE (id=5):(Total: 35m28s, non-child: 3.300us, % non-child: 0.00%) BytesReceived(1m4s): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 0 - ConvertRowBatchTime: 926.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 35m28s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F02:(Total: 35m28s, non-child: 4s804ms, % non-child: 0.23%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:35m28s max:35m28s mean: 35m28s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 1.19 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.12 GB (8716195362) - PerHostPeakMemUsage: 9.65 GB (10362634827) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 5m8s - TotalNetworkSendTime: 130.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 21.57K (21566) - TotalThreadsTotalWallClockTime: 42m22s - TotalThreadsSysTime: 15s736ms - TotalThreadsUserTime: 4m47s - TotalThreadsVoluntaryContextSwitches: 666.30K (666301) Fragment Instance Lifecycle Timings: - ExecTime: 28m34s - ExecTreeExecTime: 28m29s - OpenTime: 6m54s - ExecTreeOpenTime: 6m54s - PrepareTime: 20.993ms - ExecTreePrepareTime: 611.240us DataStreamSender (dst_id=5):(Total: 4.051us, non-child: 4.051us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 197.881ms, non-child: 197.881ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 63.528ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 112 (112) - NumInstructions: 4.99K (4991) - OptimizationTime: 116.235ms - PeakMemoryUsage: 2.44 MB (2555392) - PrepareTime: 16.992ms HASH_JOIN_NODE (id=2):(Total: 35m23s, non-child: 30m9s, % non-child: 85.21%) - BuildRows: 360.01M (360011594) - BuildTime: 4m44s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.01 GB (8599796313) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 5m29s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Hash Join Builder (join_node_id=2): - BuildRowsPartitionTime: 4m36s - BuildRowsPartitioned: 360.01M (360011594) - GetNewBlockTime: 6m43s - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 9m48s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 8.01 GB (8599412864) - PinTime: 18m40s - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) - UnpinTime: 7.670ms EXCHANGE_NODE (id=4):(Total: 2m3s, non-child: 2m3s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s103ms - DeserializeRowBatchTimer: 1m16s - FirstBatchArrivalWaitTime: 114.743ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.91 M/sec - SendersBlockedTimer: 2m43s - SendersBlockedTotalTimer(*): 2m43s EXCHANGE_NODE (id=3):(Total: 3m11s, non-child: 3m11s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s047ms - DeserializeRowBatchTimer: 1m19s - FirstBatchArrivalWaitTime: 1s376ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.88 M/sec - SendersBlockedTimer: 10m10s - SendersBlockedTotalTimer(*): 10m10s Fragment F02: Instance a540d26d367b2564:c64481bc00000003 (host=tarmstrong-box2:22000):(Total: 35m28s, non-child: 4s804ms, % non-child: 0.23%) MemoryUsage(1m4s): 6.58 GB, 7.38 GB, 6.87 GB, 6.74 GB, 5.61 GB, 5.16 GB, 6.35 GB, 6.85 GB, 6.84 GB, 6.85 GB, 6.84 GB, 6.84 GB, 6.84 GB, 4.49 GB, 5.73 GB, 5.29 GB, 4.05 GB, 4.98 GB, 5.56 GB, 5.63 GB, 4.64 GB, 4.62 GB, 5.27 GB, 6.08 GB, 4.50 GB, 4.23 GB, 5.29 GB, 5.70 GB, 5.04 GB, 4.77 GB, 4.57 GB, 5.93 GB, 4.96 GB ThreadUsage(1m4s): 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.19 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.12 GB (8716195362) - PerHostPeakMemUsage: 9.65 GB (10362634827) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 5m8s - TotalNetworkSendTime: 130.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 21.57K (21566) - TotalThreadsTotalWallClockTime: 42m22s - TotalThreadsSysTime: 15s736ms - TotalThreadsUserTime: 4m47s - TotalThreadsVoluntaryContextSwitches: 666.30K (666301) Fragment Instance Lifecycle Timings: - ExecTime: 28m34s - ExecTreeExecTime: 28m29s - OpenTime: 6m54s - ExecTreeOpenTime: 6m54s - PrepareTime: 20.993ms - ExecTreePrepareTime: 611.240us DataStreamSender (dst_id=5):(Total: 4.051us, non-child: 4.051us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 197.881ms, non-child: 197.881ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 63.528ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 112 (112) - NumInstructions: 4.99K (4991) - OptimizationTime: 116.235ms - PeakMemoryUsage: 2.44 MB (2555392) - PrepareTime: 16.992ms HASH_JOIN_NODE (id=2):(Total: 35m23s, non-child: 30m9s, % non-child: 85.21%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 360.01M (360011594) - BuildTime: 4m44s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.01 GB (8599796313) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 5m29s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled, Spilled Runtime filters: 0 of 4 Runtime Filters Published, 4 Disabled - BuildRowsPartitionTime: 4m36s - BuildRowsPartitioned: 360.01M (360011594) - GetNewBlockTime: 6m43s - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 9m48s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 8.01 GB (8599412864) - PinTime: 18m40s - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) - UnpinTime: 7.670ms EXCHANGE_NODE (id=4):(Total: 2m3s, non-child: 9s440ms, % non-child: 7.64%) BytesReceived(8s000ms): 641.83 MB, 2.04 GB, 3.44 GB, 4.55 GB, 5.44 GB, 6.19 GB, 6.91 GB, 7.62 GB, 8.41 GB, 9.43 GB, 10.09 GB, 10.69 GB, 11.55 GB, 12.40 GB, 13.21 GB, 13.93 GB, 14.76 GB, 15.45 GB, 16.20 GB, 16.98 GB, 17.62 GB, 18.42 GB, 19.00 GB, 19.87 GB, 20.59 GB, 21.31 GB, 22.06 GB, 22.78 GB, 23.35 GB, 24.12 GB, 24.90 GB, 25.72 GB, 26.55 GB, 27.27 GB, 27.84 GB, 28.46 GB, 29.36 GB, 30.16 GB, 30.97 GB, 31.77 GB, 32.45 GB, 33.19 GB, 33.96 GB, 34.70 GB, 35.16 GB, 35.82 GB, 36.72 GB, 37.34 GB, 38.04 GB, 38.72 GB, 39.59 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s103ms - DeserializeRowBatchTimer: 1m16s - FirstBatchArrivalWaitTime: 114.743ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.91 M/sec - SendersBlockedTimer: 2m43s - SendersBlockedTotalTimer(*): 2m43s EXCHANGE_NODE (id=3):(Total: 3m11s, non-child: 7s821ms, % non-child: 4.09%) BytesReceived(1m4s): 9.99 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 904.84 MB, 6.28 GB, 11.46 GB, 16.77 GB, 22.41 GB, 27.85 GB, 33.92 GB, 39.20 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s047ms - DeserializeRowBatchTimer: 1m19s - FirstBatchArrivalWaitTime: 1s376ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.88 M/sec - SendersBlockedTimer: 10m10s - SendersBlockedTotalTimer(*): 10m10s Averaged Fragment F00:(Total: 14m24s, non-child: 10m1s, % non-child: 69.59%) split sizes: min: 12.56 GB, max: 12.56 GB, avg: 12.56 GB, stddev: 0 completion times: min:14m24s max:14m24s mean: 14m24s stddev:0.000ns execution rates: min:14.87 MB/sec max:14.87 MB/sec mean:14.87 MB/sec stddev:0.60 B/sec num instances: 1 - AverageThreadTokens: 5.47 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.07 GB (1149369119) - PerHostPeakMemUsage: 9.65 GB (10362634827) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 10m15s - TotalStorageWaitTime: 2m16s - TotalThreadsInvoluntaryContextSwitches: 19.98K (19978) - TotalThreadsTotalWallClockTime: 1h18m - TotalThreadsSysTime: 1s968ms - TotalThreadsUserTime: 5m19s - TotalThreadsVoluntaryContextSwitches: 353.74K (353738) Fragment Instance Lifecycle Timings: - ExecTime: 14m24s - ExecTreeExecTime: 12s925ms - OpenTime: 46.145ms - ExecTreeOpenTime: 24.248us - PrepareTime: 18.083ms - ExecTreePrepareTime: 101.109us DataStreamSender (dst_id=3):(Total: 4m10s, non-child: 4m10s, % non-child: 100.00 - ExecTreePrepareTime: 101.109us DataStreamSender (dst_id=3):(Total: 4m10s, non-child: 4m10s, % non-child: 100.00%) - BytesSent: 39.92 GB (42858904369) - NetworkThroughput(*): 2.69 GB/sec - OverallThroughput: 163.48 MB/sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 3m55s - TransmitDataRPCTime: 14s863ms - UncompressedRowBatchSize: 91.46 GB (98207171210) CodeGen:(Total: 63.464ms, non-child: 63.464ms, % non-child: 100.00%) - CodegenTime: 932.590us - CompileTime: 7.195ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 36 (36) - NumInstructions: 655 (655) - OptimizationTime: 38.433ms - PeakMemoryUsage: 327.50 KB (335360) - PrepareTime: 17.412ms HDFS_SCAN_NODE (id=0):(Total: 12s821ms, non-child: 12s821ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 51 out of 51 Runtime filters: Not all filters arrived (arrived: [], missing [0, 1, 2, 3]), waited for 1s003ms Hdfs Read Thread Concurrency Bucket: 0:79.58% 1:3.239% 2:1.793% 3:1.504% 4:5.957% 5:2.256% 6:5.668% File Formats: PARQUET/SNAPPY:816 BytesRead(16s000ms): 808.79 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 989.96 MB, 1.00 GB, 1.12 GB, 1.81 GB, 2.10 GB, 2.60 GB, 3.02 GB, 3.15 GB, 3.85 GB, 4.09 GB, 4.75 GB, 5.05 GB, 5.29 GB, 5.97 GB, 6.08 GB, 6.87 GB, 7.08 GB, 7.78 GB, 8.02 GB, 8.41 GB, 8.94 GB, 9.24 GB, 9.93 GB, 10.28 GB, 10.94 GB, 11.16 GB, 11.93 GB, 12.12 GB, 12.52 GB - FooterProcessingTime: (Avg: 446.298ms ; Min: 976.045us ; Max: 2s801ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 0.89 - AverageScannerThreadConcurrency: 4.47 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s218ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.07 GB (1149160240) - PerReadThreadRawHdfsThroughput: 16.85 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 11s124ms - RowBatchQueuePutWaitTime: 1h - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 28.08 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 14.33K (14331) - ScannerThreadsTotalWallClockTime: 1h4m - MaterializeTupleTime(*): 55s489ms - ScannerThreadsSysTime: 1s192ms - Scanne CodegenTime: 484.159us - CompileTime: 7.667ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 6 (6) - NumInstructions: 98 (98) - OptimizationTime: 8.329ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 13.681ms HDFS_SCAN_NODE (id=1):(Total: 2s271ms, non-child: 2s271ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.44 - AverageScannerThreadConcurrency: 2.16 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s661ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.12 GB (1199387973) - PerReadThreadRawHdfsThroughput: 71.88 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 1s575ms - RowBatchQueuePutWaitTime: 13m5s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 158.47 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 21.41K (21413) - ScannerThreadsTotalWallClockTime: 14m39s - MaterializeTupleTime(*): 50s432ms - ScannerThreadsSysTime: 1s188ms - ScannerThreadsUserTime: 1m18s - ScannerThreadsVoluntaryContextSwitches: 351.26K (351259) - TotalRawHdfsReadTime(*): 2m58s - TotalReadThroughput: 31.64 MB/sec Fragment F01: Instance a540d26d367b2564:c64481bc00000002 (host=tarmstrong-box2:22000):(Total: 6m46s, non-child: 2m30s, % non-child: 36.94%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB MemoryUsage(8s000ms): 923.11 MB, 857.30 MB, 461.22 MB, 382.86 MB, 313.55 MB, 366.10 MB, 246.71 MB, 386.14 MB, 319.29 MB, 349.94 MB, 302.37 MB, 402.97 MB, 272.45 MB, 382.51 MB, 320.21 MB, 385.07 MB, 274.26 MB, 394.95 MB, 253.54 MB, 403.39 MB, 283.41 MB, 405.54 MB, 320.40 MB, 373.59 MB, 324.88 MB, 333.23 MB, 346.14 MB, 324.18 MB, 389.52 MB, 298.67 MB, 397.65 MB, 255.51 MB, 385.54 MB, 279.32 MB, 421.81 MB, 321.20 MB, 326.67 MB, 290.33 MB, 381.26 MB, 275.78 MB, 363.86 MB, 284.60 MB, 350.78 MB, 284.49 MB, 295.77 MB, 379.63 MB, 336.66 MB, 382.65 MB, 267.68 MB, 393.47 MB ThreadUsage(8s000ms): 6, 6, 5, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3 - AverageThreadTokens: 3.16 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1199595743) - PerHostPeakMemUsage: 9.65 GB (10362634827) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 2m45s - TotalStorageWaitTime: 20s812ms - TotalThreadsInvoluntaryContextSwitches: 27.63K (27632) - TotalThreadsTotalWallClockTime: 21m26s - TotalThreadsSysTime: 2s028ms - TotalThreadsUserTime: 5m16s - TotalThreadsVoluntaryContextSwitches: 353.29K (353294) GB, 4.15 GB, 4.35 GB, 4.64 GB, 4.78 GB, 5.13 GB, 5.21 GB, 5.62 GB, 5.67 GB, 6.09 GB, 6.15 GB, 6.52 GB, 6.65 GB, 6.93 GB, 7.14 GB, 7.37 GB, 7.61 GB, 7.76 GB, 8.10 GB, 8.21 GB, 8.60 GB, 8.72 GB, 9.07 GB, 9.12 GB, 9.45 GB, 9.64 GB, 10.00 GB, 10.13 GB, 10.44 GB, 10.59 GB, 10.91 GB, 11.06 GB, 11.23 GB, 11.51 GB, 11.77 GB, 12.00 GB, 12.10 GB, 12.46 GB - FooterProcessingTime: (Avg: 42.764ms ; Min: 1.700ms ; Max: 165.235ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 0.44 - AverageScannerThreadConcurrency: 2.16 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s661ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.12 GB (1199387973) - PerReadThreadRawHdfsThroughput: 71.88 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 1s575ms - RowBatchQueuePutWaitTime: 13m5s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 158.47 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 21.41K (21413) - ScannerThreadsTotalWallClockTime: 14m39s - MaterializeTupleTime(*): 50s432ms - ScannerThreadsSysTime: 1s188ms - ScannerThreadsUserTime: 1m18s - ScannerThreadsVoluntaryContextSwitches: 351.26K (351259) - TotalRawHdfsReadTime(*): 2m58s - TotalReadThroughput: 31.64 MB/sec Query (id=974c7375a197bc33:1e908bfe00000000): Summary: Session ID: 2e472a715692f619:dd5dd64066c604aa Session Type: BEESWAX Start Time: 2017-08-17 20:26:49.461646000 End Time: 2017-08-17 21:04:33.782111000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.10.0-SNAPSHOT RELEASE (build d5b0c6b9366bc54801ac0dc74c942cdbda213b67) User: tarmstrong Connected User: tarmstrong Delegated User: Network Address: ::ffff:127.0.0.1:60092 Default Db: tpch_60_parquet Sql Statement: select * from lineitem l1 join lineitem l2 on l1.l_orderkey = l2.l_orderkey and l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey and l1.l_linenumber = l2.l_linenumber where l1.l_tax < l2.l_tax Coordinator: tarmstrong-box2:22000 Query Options (non default): MEM_LIMIT=10737418240,MT_DOP=0 Plan: ---------------- Per-Host Resource Reservation: Memory=136.00MB Per-Host Resource Estimates: Memory=99.77GB F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=0B mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 05:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=0,1 row-size=534B cardinality=360011594 | F02:PLAN FRAGMENT [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] hosts=1 instances=1 Per-Host Resources: mem-estimate=98.39GB mem-reservation=136.00MB 02:HASH JOIN [INNER JOIN, PARTITIONED] | hash predicates: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | fk/pk conjuncts: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | other predicates: l1.l_tax < l2.l_tax | runtime filters: RF000 <- l2.l_linenumber, RF001 <- l2.l_orderkey, RF002 <- l2.l_partkey, RF003 <- l2.l_suppkey | mem-estimate=98.39GB mem-reservation=136.00MB | tuple-ids=0,1 row-size=534B cardinality=360011594 | |--04:EXCHANGE [HASH(l2.l_linenumber,l2.l_orderkey,l2.l_partkey,l2.l_suppkey)] | | mem-estimate=0B mem-reservation=0B | | tuple-ids=1 row-size=267B cardinality=360011594 | | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 | Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B | 01:SCAN HDFS [tpch_60_parquet.lineitem l2, RANDOM] | partitions=1/1 files=51 size=12.56GB | stats-rows=360011594 extrapolated-rows=disabled | table stats: rows=360011594 size=12.56GB | column stats: all | mem-estimate=704.00MB mem-reservation=0B | tuple-ids=1 row-size=267B cardinality=360011594 | 03:EXCHANGE [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] | mem-estimate=0B mem-reservation=0B | tuple-ids=0 row-size=267B cardinality=360011594 | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B 00:SCAN HDFS [tpch_60_parquet.lineitem l1, RANDOM] partitions=1/1 files=51 size=12.56GB runtime filters: RF000 -> l1.l_linenumber, RF001 -> l1.l_orderkey, RF002 -> l1.l_partkey, RF003 -> l1.l_suppkey stats-rows=360011594 extrapolated-rows=disabled table stats: rows=360011594 size=12.56GB column stats: all mem-estimate=704.00MB mem-reservation=0B tuple-ids=0 row-size=267B cardinality=360011594 ---------------- Estimated Per-Host Mem: 107126659482 Per-Host Memory Reservation: 142606336 Request Pool: default-pool Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail --------------------------------------------------------------------------------------------------------------------------- 05:EXCHANGE 1 4.139us 4.139us 0 360.01M 0 0 UNPARTITIONED 02:HASH JOIN 1 29m39s 29m39s 0 360.01M 8.01 GB 98.39 GB INNER JOIN, PARTITIONED |--04:EXCHANGE 1 7s690ms 7s690ms 360.01M 360.01M 0 0 HASH(l2.l_linenumber,l2.l_o... | 01:SCAN HDFS 1 2m9s 2m9s 360.01M 360.01M 1.11 GB 704.00 MB tpch_60_parquet.lineitem l2 03:EXCHANGE 1 7s549ms 7s549ms 360.01M 360.01M 0 0 HASH(l1.l_linenumber,l1.l_o... 00:SCAN HDFS 1 47s422ms 47s422ms 360.01M 360.01M 1.12 GB 704.00 MB tpch_60_parquet.lineitem l1 Errors: Planner Timeline: 5s378ms - Metadata load started: 32.579ms (32.579ms) - Metadata load finished: 5s272ms (5s240ms) - Analysis finished: 5s279ms (6.916ms) - Equivalence classes computed: 5s296ms (16.737ms) - Single node plan created: 5s327ms (31.310ms) - Runtime filters computed: 5s331ms (3.316ms) - Distributed plan created: 5s332ms (1.632ms) - Planning finished: 5s378ms (45.730ms) Query Timeline: 37m44s - Query submitted: 16.882us (16.882us) - Planning finished: 5s388ms (5s388ms) - Submit for admission: 5s388ms (164.854us) - Completed admission: 5s389ms (52.709us) - Ready to start on 1 backends: 5s389ms (121.965us) - All 1 execution backends (4 fragment instances) started: 5s389ms (815.729us) - First dynamic filter received: 8m16s (8m11s) - Rows available: 37m43s (29m27s) - First row fetched: 37m44s (649.207ms) - Unregister query: 37m44s (3.358ms) - ComputeScanRangeAssignmentTimer: 61.492us ImpalaServer: - ClientFetchWaitTimer: 651.203ms - RowMaterializationTimer: 1.327ms Execution Profile 974c7375a197bc33:1e908bfe00000000:(Total: 37m38s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 4 Filter routing table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 1 (1) N/A N/A true 2 2 0 REMOTE false 1 (1) N/A N/A true 1 2 0 REMOTE false 1 (1) N/A N/A true 0 2 0 REMOTE false 1 (1) N/A N/A true Backend startup latencies: Count: 1, min / max: 0 / 0, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 0, 99.9th %-ile: 0 Final filter table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 0 (1) 8m16s 8m16s false 2 2 0 REMOTE false 0 (1) 8m16s 8m16s false 1 2 0 REMOTE false 0 (1) 8m16s 8m16s false 0 2 0 REMOTE false 0 (1) 8m16s 8m16s false Per Node Peak Memory Usage: tarmstrong-box2:22000(9.68 GB) - FiltersReceived: 4 (4) - FinalizationTimer: 0.000ns Averaged Fragment F03:(Total: 37m38s, non-child: 262.119us, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:37m38s max:37m38s mean: 37m38s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PerHostPeakMemUsage: 9.68 GB (10389722123) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 37m38s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 37m38s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 6.454us - ExecTreeExecTime: 2.463us - OpenTime: 37m38s - ExecTreeOpenTime: 37m38s - PrepareTime: 10.692ms - ExecTreePrepareTime: 34.303us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 22.39K (22395) - BlocksRecycled: 15 (15) - BufferedPins: 170 (170) - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 8.00 GB (8589934592) - PeakMemoryUsage: 8.00 GB (8589934592) - ScratchBytesRead: 167.74 GB (180111802368) - ScratchBytesWritten: 167.74 GB (180111802368) - ScratchFileUsedBytes: 167.74 GB (180111802368) - ScratchReads: 21.47K (21471) - ScratchWrites: 21.47K (21471) - TotalBufferWaitTime: 7m20s - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 17m41s PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 10.481ms, non-child: 10.481ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 10.164ms EXCHANGE_NODE (id=5):(Total: 37m38s, non-child: 37m38s, % non-child: 100.00%) - BytesReceived: 0 - ConvertRowBatchTime: 631.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 37m38s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Coordinator Fragment F03: Instance 974c7375a197bc33:1e908bfe00000000 (host=tarmstrong-box2:22000):(Total: 37m38s, non-child: 262.119us, % non-child: 0.00%) MemoryUsage(1m4s): 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PerHostPeakMemUsage: 9.68 GB (10389722123) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 37m38s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 37m38s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 6.454us - ExecTreeExecTime: 2.463us - OpenTime: 37m38s - ExecTreeOpenTime: 37m38s - PrepareTime: 10.692ms - ExecTreePrepareTime: 34.303us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 22.39K (22395) - BlocksRecycled: 15 (15) - BufferedPins: 170 (170) - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 8.00 GB (8589934592) - PeakMemoryUsage: 8.00 GB (8589934592) - ScratchBytesRead: 167.74 GB (180111802368) - ScratchBytesWritten: 167.74 GB (180111802368) - ScratchFileUsedBytes: 167.74 GB (180111802368) - ScratchReads: 21.47K (21471) - ScratchWrites: 21.47K (21471) - TotalBufferWaitTime: 7m20s - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 17m41s PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 10.481ms, non-child: 10.481ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 10.164ms EXCHANGE_NODE (id=5):(Total: 37m38s, non-child: 4.139us, % non-child: 0.00%) BytesReceived(1m4s): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 0 - ConvertRowBatchTime: 631.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 37m38s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F02:(Total: 37m38s, non-child: 4s777ms, % non-child: 0.21%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:37m38s max:37m38s mean: 37m38s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 1.22 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.12 GB (8716198177) - PerHostPeakMemUsage: 9.68 GB (10389722123) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 7m48s - TotalNetworkSendTime: 137.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 17.91K (17907) - TotalThreadsTotalWallClockTime: 45m55s - TotalThreadsSysTime: 16s836ms - TotalThreadsUserTime: 4m40s - TotalThreadsVoluntaryContextSwitches: 678.91K (678913) Fragment Instance Lifecycle Timings: - ExecTime: 29m18s - ExecTreeExecTime: 29m13s - OpenTime: 8m19s - ExecTreeOpenTime: 8m19s - PrepareTime: 15.776ms - ExecTreePrepareTime: 58.525us DataStreamSender (dst_id=5):(Total: 5.614us, non-child: 5.614us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 167.478ms, non-child: 167.478ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 47.072ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 112 (112) - NumInstructions: 4.99K (4991) - OptimizationTime: 107.155ms - PeakMemoryUsage: 2.44 MB (2555392) - PrepareTime: 12.952ms HASH_JOIN_NODE (id=2):(Total: 37m33s, non-child: 29m39s, % non-child: 78.97%) - BuildRows: 360.01M (360011594) - BuildTime: 3m56s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.01 GB (8599746950) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 6m44s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Hash Join Builder (join_node_id=2): - BuildRowsPartitionTime: 3m49s - BuildRowsPartitioned: 360.01M (360011594) - GetNewBlockTime: 7m17s - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 9m27s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 8.01 GB (8599412864) - PinTime: 17m41s - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) - UnpinTime: 6.557ms EXCHANGE_NODE (id=4):(Total: 4m15s, non-child: 4m15s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 2s857ms - DeserializeRowBatchTimer: 1m15s - FirstBatchArrivalWaitTime: 240.069ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.41 M/sec - SendersBlockedTimer: 2m7s - SendersBlockedTotalTimer(*): 2m7s EXCHANGE_NODE (id=3):(Total: 3m39s, non-child: 3m39s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s054ms - DeserializeRowBatchTimer: 1m21s - FirstBatchArrivalWaitTime: 1s485ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.64 M/sec - SendersBlockedTimer: 12m41s - SendersBlockedTotalTimer(*): 12m41s Fragment F02: Instance 974c7375a197bc33:1e908bfe00000003 (host=tarmstrong-box2:22000):(Total: 37m38s, non-child: 4s777ms, % non-child: 0.21%) MemoryUsage(1m4s): 6.55 GB, 7.38 GB, 6.77 GB, 6.81 GB, 6.12 GB, 4.74 GB, 5.27 GB, 6.01 GB, 6.84 GB, 6.85 GB, 6.84 GB, 6.85 GB, 6.85 GB, 6.85 GB, 6.84 GB, 6.84 GB, 4.77 GB, 5.15 GB, 5.91 GB, 4.69 GB, 4.66 GB, 4.84 GB, 5.09 GB, 5.84 GB, 5.06 GB, 5.00 GB, 4.54 GB, 4.81 GB, 5.59 GB, 5.07 GB, 4.72 GB, 4.53 GB, 5.40 GB, 5.90 GB, 4.57 GB ThreadUsage(1m4s): 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.22 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.12 GB (8716198177) - PerHostPeakMemUsage: 9.68 GB (10389722123) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 7m48s - TotalNetworkSendTime: 137.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 17.91K (17907) - TotalThreadsTotalWallClockTime: 45m55s - TotalThreadsSysTime: 16s836ms - TotalThreadsUserTime: 4m40s - TotalThreadsVoluntaryContextSwitches: 678.91K (678913) Fragment Instance Lifecycle Timings: - ExecTime: 29m18s - ExecTreeExecTime: 29m13s - OpenTime: 8m19s - ExecTreeOpenTime: 8m19s - PrepareTime: 15.776ms - ExecTreePrepareTime: 58.525us DataStreamSender (dst_id=5):(Total: 5.614us, non-child: 5.614us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 167.478ms, non-child: 167.478ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 47.072ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 112 (112) - NumInstructions: 4.99K (4991) - OptimizationTime: 107.155ms - PeakMemoryUsage: 2.44 MB (2555392) - PrepareTime: 12.952ms HASH_JOIN_NODE (id=2):(Total: 37m33s, non-child: 29m39s, % non-child: 78.97%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 360.01M (360011594) - BuildTime: 3m56s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.01 GB (8599746950) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 6m44s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled, Spilled Runtime filters: 0 of 4 Runtime Filters Published, 4 Disabled - BuildRowsPartitionTime: 3m49s - BuildRowsPartitioned: 360.01M (360011594) - GetNewBlockTime: 7m17s - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 9m27s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 8.01 GB (8599412864) - PinTime: 17m41s - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) - UnpinTime: 6.557ms EXCHANGE_NODE (id=4):(Total: 4m15s, non-child: 7s690ms, % non-child: 3.01%) BytesReceived(8s000ms): 644.01 MB, 2.05 GB, 3.45 GB, 4.52 GB, 5.39 GB, 6.22 GB, 6.91 GB, 7.58 GB, 8.16 GB, 9.02 GB, 9.78 GB, 10.68 GB, 11.36 GB, 12.21 GB, 12.96 GB, 13.79 GB, 14.65 GB, 15.34 GB, 16.15 GB, 16.84 GB, 17.55 GB, 18.45 GB, 19.44 GB, 20.16 GB, 20.73 GB, 21.49 GB, 22.31 GB, 23.09 GB, 23.87 GB, 24.41 GB, 25.10 GB, 25.76 GB, 26.24 GB, 26.55 GB, 27.15 GB, 27.64 GB, 28.09 GB, 28.66 GB, 29.25 GB, 29.75 GB, 30.27 GB, 30.86 GB, 31.33 GB, 31.91 GB, 32.36 GB, 32.63 GB, 33.16 GB, 33.67 GB, 34.05 GB, 34.56 GB, 35.13 GB, 35.62 GB, 36.03 GB, 36.62 GB, 37.02 GB, 37.30 GB, 37.79 GB, 38.04 GB, 38.62 GB, 38.91 GB, 39.49 GB, 39.87 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 2s857ms - DeserializeRowBatchTimer: 1m15s - FirstBatchArrivalWaitTime: 240.069ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.41 M/sec - SendersBlockedTimer: 2m7s - SendersBlockedTotalTimer(*): 2m7s EXCHANGE_NODE (id=3):(Total: 3m39s, non-child: 7s549ms, % non-child: 3.45%) BytesReceived(1m4s): 9.99 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 139.77 MB, 3.54 GB, 7.48 GB, 11.03 GB, 14.43 GB, 18.03 GB, 21.92 GB, 27.37 GB, 33.68 GB, 39.27 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s054ms - DeserializeRowBatchTimer: 1m21s - FirstBatchArrivalWaitTime: 1s485ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 1.64 M/sec - SendersBlockedTimer: 12m41s - SendersBlockedTotalTimer(*): 12m41s Averaged Fragment F00:(Total: 17m33s, non-child: 12m35s, % non-child: 71.65%) split sizes: min: 12.56 GB, max: 12.56 GB, avg: 12.56 GB, stddev: 0 completion times: min:17m33s max:17m33s mean: 17m33s stddev:0.000ns execution rates: min:12.20 MB/sec max:12.20 MB/sec mean:12.20 MB/sec stddev:0.44 B/sec num instances: 1 - AverageThreadTokens: 5.50 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1199446788) - PerHostPeakMemUsage: 9.68 GB (10389722123) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 12m48s - TotalStorageWaitTime: 7m10s - TotalThreadsInvoluntaryContextSwitches: 13.82K (13816) - TotalThreadsTotalWallClockTime: 1h36m - TotalThreadsSysTime: 1s888ms - TotalThreadsUserTime: 5m18s - TotalThreadsVoluntaryContextSwitches: 352.69K (352694) Fragment Instance Lifecycle Timings: - ExecTime: 17m33s kMemUsage: 9.68 GB (10389722123) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 12m48s - TotalStorageWaitTime: 7m10s - TotalThreadsInvoluntaryContextSwitches: 13.82K (13816) - TotalThreadsTotalWallClockTime: 1h36m - TotalThreadsSysTime: 1s888ms - TotalThreadsUserTime: 5m18s - TotalThreadsVoluntaryContextSwitches: 352.69K (352694) Fragment Instance Lifecycle Timings: - ExecTime: 17m33s - ExecTreeExecTime: 47s491ms - OpenTime: 46.221ms - ExecTreeOpenTime: 22.235us - PrepareTime: 30.141ms - ExecTreePrepareTime: 12.629ms DataStreamSender (dst_id=3):(Total: 4m11s, non-child: 4m11s, % non-child: 100.00%) - BytesSent: 39.92 GB (42858904369) - NetworkThroughput(*): 2.74 GB/sec - OverallThroughput: 162.67 MB/sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 3m56s - TransmitDataRPCTime: 14s579ms - UncompressedRowBatchSize: 91.46 GB (98207171210) CodeGen:(Total: 62.656ms, non-child: 62.656ms, % non-child: 100.00%) - CodegenTime: 1.296ms - CompileTime: 7.229ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 36 (36) - NumInstructions: 655 (655) - OptimizationTime: 38.379ms - PeakMemoryUsage: 327.50 KB (335360) - PrepareTime: 16.597ms HDFS_SCAN_NODE (id=0):(Total: 47s422ms, non-child: 47s422ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 51 out of 51 Runtime filters: Not all filters arrived (arrived: [], missing [0, 1, 2, 3]), waited for 1s004ms Hdfs Read Thread Concurrency Bucket: 0:64.31% 1:2.943% 2:3.275% 3:3.844% 4:10.49% 5:1.804% 6:13.34% File Formats: PARQUET/SNAPPY:816 BytesRead(32s000ms): 934.16 MB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.01 GB, 1.14 GB, 1.79 GB, 2.22 GB, 3.06 GB, 3.37 GB, 4.08 GB, 4.57 GB, 5.07 GB, 5.74 GB, 6.32 GB, 6.82 GB, 7.63 GB, 8.51 GB, 9.42 GB, 10.45 GB, 11.50 GB - FooterProcessingTime: (Avg: 1s567ms ; Min: 1.229ms ; Max: 9s880ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 1.75 - AverageScannerThreadConcurrency: 4.50 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 18s953ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.12 GB (1199313436) - PerReadThreadRawHdfsThroughput: 6.95 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 45s705ms -MemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 3m50s - TransmitDataRPCTime: 10s856ms - UncompressedRowBatchSize: 91.46 GB (98207171210) CodeGen:(Total: 26.139ms, non-child: 26.139ms, % non-child: 100.00%) - CodegenTime: 399.007us - CompileTime: 4.229ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2049660) - NumFunctions: 6 (6) - NumInstructions: 98 (98) - OptimizationTime: 5.317ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 16.144ms HDFS_SCAN_NODE (id=1):(Total: 2m9s, non-child: 2m9s, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.68 - AverageScannerThreadConcurrency: 1.18 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s124ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.11 GB (1195155083) - PerReadThreadRawHdfsThroughput: 38.55 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 2m9s - RowBatchQueuePutWaitTime: 6m16s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.77 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 7.38K (7382) - ScannerThreadsTotalWallClockTime: 9m40s - MaterializeTupleTime(*): 45s875ms - ScannerThreadsSysTime: 1s328ms - ScannerThreadsUserTime: 1m13s - ScannerThreadsVoluntaryContextSwitches: 344.89K (344888) - TotalRawHdfsReadTime(*): 5m33s - TotalReadThroughput: 26.20 MB/sec Fragment F01: Instance 974c7375a197bc33:1e908bfe00000002 (host=tarmstrong-box2:22000):(Total: 8m11s, non-child: 2m, % non-child: 24.44%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB MemoryUsage(8s000ms): 901.72 MB, 857.35 MB, 438.57 MB, 177.56 MB, 160.97 MB, 175.17 MB, 168.18 MB, 157.38 MB, 171.72 MB, 152.84 MB, 192.94 MB, 176.87 MB, 183.02 MB, 156.66 MB, 186.61 MB, 163.08 MB, 150.08 MB, 182.93 MB, 164.83 MB, 171.40 MB, 163.32 MB, 155.83 MB, 177.63 MB, 155.26 MB, 145.40 MB, 175.04 MB, 150.09 MB, 171.71 MB, 170.39 MB, 130.82 MB, 171.48 MB, 141.83 MB, 194.71 MB, 95.48 MB, 129.35 MB, 193.96 MB, 71.64 MB, 156.68 MB, 166.40 MB, 181.45 MB, 131.28 MB, 152.95 MB, 96.23 MB, 164.55 MB, 179.88 MB, 101.95 MB, 167.44 MB, 102.46 MB, 155.52 MB, 145.48 MB, 84.91 MB, 143.00 MB, 80.34 MB, 91.26 MB, 153.42 MB, 117.33 MB, 141.53 MB, 158.67 MB, 202.10 MB, 135.20 MB, 168.19 MB ThreadUsage(8s000ms): 6, 6, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 - AverageThreadTokens: 2.18 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.11 GB (1195390445) - PerHostPeakMemUsage: 9.68 GB (10389722123) - Row:30.04% 2:1.629% 3:0.1018% 4:1.426% 5:0.1018% 6:3.666% File Formats: PARQUET/SNAPPY:816 BytesRead(8s000ms): 877.12 MB, 1.13 GB, 1.26 GB, 1.53 GB, 1.78 GB, 2.06 GB, 2.27 GB, 2.46 GB, 2.66 GB, 2.92 GB, 3.20 GB, 3.46 GB, 3.68 GB, 3.94 GB, 4.18 GB, 4.43 GB, 4.69 GB, 4.94 GB, 5.18 GB, 5.40 GB, 5.61 GB, 5.89 GB, 6.23 GB, 6.42 GB, 6.60 GB, 6.87 GB, 7.10 GB, 7.36 GB, 7.61 GB, 7.76 GB, 8.01 GB, 8.17 GB, 8.38 GB, 8.40 GB, 8.60 GB, 8.82 GB, 8.85 GB, 9.10 GB, 9.30 GB, 9.47 GB, 9.57 GB, 9.79 GB, 9.92 GB, 10.13 GB, 10.29 GB, 10.32 GB, 10.53 GB, 10.66 GB, 10.79 GB, 10.95 GB, 11.07 GB, 11.28 GB, 11.39 GB, 11.56 GB, 11.74 GB, 11.80 GB, 11.97 GB, 12.06 GB, 12.28 GB, 12.29 GB, 12.52 GB - FooterProcessingTime: (Avg: 629.888ms ; Min: 1.945ms ; Max: 4s300ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 0.68 - AverageScannerThreadConcurrency: 1.18 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 19s124ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.11 GB (1195155083) - PerReadThreadRawHdfsThroughput: 38.55 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 2m9s - RowBatchQueuePutWaitTime: 6m16s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.77 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 7.38K (7382) - ScannerThreadsTotalWallClockTime: 9m40s - MaterializeTupleTime(*): 45s875ms - ScannerThreadsSysTime: 1s328ms - ScannerThreadsUserTime: 1m13s - ScannerThreadsVoluntaryContextSwitches: 344.89K (344888) - TotalRawHdfsReadTime(*): 5m33s - TotalReadThroughput: 26.20 MB/sec