Query (id=d5451fbe5d241e9c:55d9edaa00000000): Summary: Session ID: f84755fd23448d06:ed727993551443af Session Type: BEESWAX Start Time: 2017-08-17 17:15:04.124099000 End Time: 2017-08-17 18:06:17.517730000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.10.0-SNAPSHOT RELEASE (build 473f717d0f38397f61f35e0ac9e32226d837a28a) User: tarmstrong Connected User: tarmstrong Delegated User: Network Address: ::ffff:127.0.0.1:55002 Default Db: tpch_60_parquet Sql Statement: select * from lineitem l1 join lineitem l2 on l1.l_orderkey = l2.l_orderkey and l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey and l1.l_linenumber = l2.l_linenumber where l1.l_tax < l2.l_tax Coordinator: tarmstrong-box2:22000 Query Options (non default): MEM_LIMIT=10737418240,MT_DOP=0 Plan: ---------------- Per-Host Resource Reservation: Memory=34.00MB Per-Host Resource Estimates: Memory=99.77GB F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=0B mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 05:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=0,1 row-size=534B cardinality=360011594 | F02:PLAN FRAGMENT [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] hosts=1 instances=1 Per-Host Resources: mem-estimate=98.39GB mem-reservation=34.00MB 02:HASH JOIN [INNER JOIN, PARTITIONED] | hash predicates: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | fk/pk conjuncts: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | other predicates: l1.l_tax < l2.l_tax | runtime filters: RF000 <- l2.l_linenumber, RF001 <- l2.l_orderkey, RF002 <- l2.l_partkey, RF003 <- l2.l_suppkey | mem-estimate=98.39GB mem-reservation=34.00MB spill-buffer=2.00MB | tuple-ids=0,1 row-size=534B cardinality=360011594 | |--04:EXCHANGE [HASH(l2.l_linenumber,l2.l_orderkey,l2.l_partkey,l2.l_suppkey)] | | mem-estimate=0B mem-reservation=0B | | tuple-ids=1 row-size=267B cardinality=360011594 | | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 | Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B | 01:SCAN HDFS [tpch_60_parquet.lineitem l2, RANDOM] | partitions=1/1 files=51 size=12.56GB | stats-rows=360011594 extrapolated-rows=disabled | table stats: rows=360011594 size=12.56GB | column stats: all | mem-estimate=704.00MB mem-reservation=0B | tuple-ids=1 row-size=267B cardinality=360011594 | 03:EXCHANGE [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] | mem-estimate=0B mem-reservation=0B | tuple-ids=0 row-size=267B cardinality=360011594 | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B 00:SCAN HDFS [tpch_60_parquet.lineitem l1, RANDOM] partitions=1/1 files=51 size=12.56GB runtime filters: RF000 -> l1.l_linenumber, RF001 -> l1.l_orderkey, RF002 -> l1.l_partkey, RF003 -> l1.l_suppkey stats-rows=360011594 extrapolated-rows=disabled table stats: rows=360011594 size=12.56GB column stats: all mem-estimate=704.00MB mem-reservation=0B tuple-ids=0 row-size=267B cardinality=360011594 ---------------- Estimated Per-Host Mem: 107126659482 Per-Host Memory Reservation: 35651584 Request Pool: default-pool Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail --------------------------------------------------------------------------------------------------------------------------- 05:EXCHANGE 1 7.337us 7.337us 0 360.01M 0 0 UNPARTITIONED 02:HASH JOIN 1 36m18s 36m18s 0 360.01M 8.00 GB 98.39 GB INNER JOIN, PARTITIONED |--04:EXCHANGE 1 7s931ms 7s931ms 360.01M 360.01M 0 0 HASH(l2.l_linenumber,l2.l_o... | 01:SCAN HDFS 1 6m30s 6m30s 360.01M 360.01M 1.11 GB 704.00 MB tpch_60_parquet.lineitem l2 03:EXCHANGE 1 11s158ms 11s158ms 360.01M 360.01M 0 0 HASH(l1.l_linenumber,l1.l_o... 00:SCAN HDFS 1 3m31s 3m31s 360.01M 360.01M 1.12 GB 704.00 MB tpch_60_parquet.lineitem l1 Errors: Planner Timeline: 5s407ms - Metadata load started: 35.156ms (35.156ms) - Metadata load finished: 5s299ms (5s263ms) - Analysis finished: 5s305ms (6.840ms) - Equivalence classes computed: 5s322ms (16.663ms) - Single node plan created: 5s353ms (31.431ms) - Runtime filters computed: 5s357ms (3.309ms) - Distributed plan created: 5s358ms (1.639ms) - Planning finished: 5s407ms (48.363ms) Query Timeline: 51m13s - Query submitted: 18.203us (18.203us) - Planning finished: 5s417ms (5s417ms) - Submit for admission: 5s417ms (170.489us) - Completed admission: 5s418ms (53.472us) - Ready to start on 1 backends: 5s418ms (117.418us) - All 1 execution backends (4 fragment instances) started: 5s419ms (901.168us) - First dynamic filter received: 11m17s (11m12s) - Rows available: 51m13s (39m55s) - First row fetched: 51m13s (277.255ms) - Unregister query: 51m13s (6.146ms) - ComputeScanRangeAssignmentTimer: 60.514us ImpalaServer: - ClientFetchWaitTimer: 281.818ms - RowMaterializationTimer: 1.547ms Execution Profile d5451fbe5d241e9c:55d9edaa00000000:(Total: 51m7s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 4 Filter routing table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 1 (1) N/A N/A true 2 2 0 REMOTE false 1 (1) N/A N/A true 1 2 0 REMOTE false 1 (1) N/A N/A true 0 2 0 REMOTE false 1 (1) N/A N/A true Backend startup latencies: Count: 1, min / max: 1ms / 1ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms Final filter table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 0 (1) 11m17s 11m17s false 2 2 0 REMOTE false 0 (1) 11m17s 11m17s false 1 2 0 REMOTE false 0 (1) 11m17s 11m17s false 0 2 0 REMOTE false 0 (1) 11m17s 11m17s false Per Node Peak Memory Usage: tarmstrong-box2:22000(9.66 GB) - FiltersReceived: 4 (4) - FinalizationTimer: 0.000ns Averaged Fragment F03:(Total: 51m7s, non-child: 283.159us, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:51m7s max:51m7s mean: 51m7s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.66 GB (10376993786) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 51m7s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 51m7s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 22.332us - ExecTreeExecTime: 5.785us - OpenTime: 51m7s - ExecTreeOpenTime: 51m7s - PrepareTime: 16.040ms - ExecTreePrepareTime: 23.131us PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 15.837ms, non-child: 15.837ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 15.439ms EXCHANGE_NODE (id=5):(Total: 51m7s, non-child: 51m7s, % non-child: 100.00%) - BytesReceived: 0 - ConvertRowBatchTime: 1.750us - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 51m7s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Coordinator Fragment F03: Instance d5451fbe5d241e9c:55d9edaa00000000 (host=tarmstrong-box2:22000):(Total: 51m7s, non-child: 283.159us, % non-child: 0.00%) MemoryUsage(1m4s): 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.66 GB (10376993786) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 51m7s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 51m7s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 22.332us - ExecTreeExecTime: 5.785us - OpenTime: 51m7s - ExecTreeOpenTime: 51m7s - PrepareTime: 16.040ms - ExecTreePrepareTime: 23.131us PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 15.837ms, non-child: 15.837ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 15.439ms EXCHANGE_NODE (id=5):(Total: 51m7s, non-child: 7.337us, % non-child: 0.00%) BytesReceived(1m4s): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 0 - ConvertRowBatchTime: 1.750us - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 51m7s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F02:(Total: 51m7s, non-child: 6s164ms, % non-child: 0.20%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:51m7s max:51m7s mean: 51m7s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 1.22 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.11 GB (8706766296) - PeakReservation: 8.00 GB (8589934592) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.66 GB (10376993786) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 14m37s - TotalNetworkSendTime: 3.036us - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 31.66K (31659) - TotalThreadsTotalWallClockTime: 1h2m - TotalThreadsSysTime: 36s908ms - TotalThreadsUserTime: 5m35s - TotalThreadsVoluntaryContextSwitches: 690.25K (690250) Fragment Instance Lifecycle Timings: - ExecTime: 39m49s - ExecTreeExecTime: 39m43s - OpenTime: 11m18s - ExecTreeOpenTime: 11m18s - PrepareTime: 17.445ms - ExecTreePrepareTime: 74.900us DataStreamSender (dst_id=5):(Total: 6.222us, non-child: 6.222us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 176.270ms, non-child: 176.270ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 46.669ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 110 (110) - NumInstructions: 5.01K (5009) - OptimizationTime: 114.499ms - PeakMemoryUsage: 2.45 MB (2564608) - PrepareTime: 14.763ms HASH_JOIN_NODE (id=2):(Total: 51m1s, non-child: 36m18s, % non-child: 71.15%) - BuildRows: 360.01M (360011594) - BuildTime: 2m40s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.00 GB (8590528921) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 5m44s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Buffer pool: - AllocTime: 3s965ms - CumulativeAllocationBytes: 357.66 GB (384038862848) - CumulativeAllocations: 174.95K (174948) - PeakReservation: 8.00 GB (8589934592) - PeakUnpinnedBytes: 169.02 GB (181479145472) - PeakUsedReservation: 8.00 GB (8589934592) - ReadIoBytes: 167.01 GB (179329564672) - ReadIoOps: 85.51K (85511) - ReadIoWaitTime: 25m52s - WriteIoBytes: 169.02 GB (181479145472) - WriteIoOps: 86.54K (86536) - WriteIoWaitTime: 4m32s Hash Join Builder (join_node_id=2): - BuildRowsPartitionTime: 2m37s - BuildRowsPartitioned: 360.01M (360011594) - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 14m54s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 48.12 KB (49280) - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) EXCHANGE_NODE (id=4):(Total: 8m29s, non-child: 8m29s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 2s820ms - DeserializeRowBatchTimer: 1m17s - FirstBatchArrivalWaitTime: 225.592ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 707.29 K/sec - SendersBlockedTimer: 46s201ms - SendersBlockedTotalTimer(*): 46s201ms EXCHANGE_NODE (id=3):(Total: 6m15s, non-child: 6m15s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s421ms - DeserializeRowBatchTimer: 1m24s - FirstBatchArrivalWaitTime: 1s469ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 959.81 K/sec - SendersBlockedTimer: 14m16s - SendersBlockedTotalTimer(*): 14m16s Fragment F02: Instance d5451fbe5d241e9c:55d9edaa00000003 (host=tarmstrong-box2:22000):(Total: 51m7s, non-child: 6s164ms, % non-child: 0.20%) MemoryUsage(1m4s): 6.73 GB, 8.10 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.09 GB, 8.08 GB, 8.07 GB, 8.07 GB, 8.07 GB, 8.07 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.07 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB ThreadUsage(1m4s): 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.22 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.11 GB (8706766296) - PeakReservation: 8.00 GB (8589934592) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.66 GB (10376993786) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 14m37s - TotalNetworkSendTime: 3.036us - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 31.66K (31659) - TotalThreadsTotalWallClockTime: 1h2m - TotalThreadsSysTime: 36s908ms - TotalThreadsUserTime: 5m35s - TotalThreadsVoluntaryContextSwitches: 690.25K (690250) Fragment Instance Lifecycle Timings: - ExecTime: 39m49s - ExecTreeExecTime: 39m43s - OpenTime: 11m18s - ExecTreeOpenTime: 11m18s - PrepareTime: 17.445ms - ExecTreePrepareTime: 74.900us DataStreamSender (dst_id=5):(Total: 6.222us, non-child: 6.222us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 176.270ms, non-child: 176.270ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 46.669ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 110 (110) - NumInstructions: 5.01K (5009) - OptimizationTime: 114.499ms - PeakMemoryUsage: 2.45 MB (2564608) - PrepareTime: 14.763ms HASH_JOIN_NODE (id=2):(Total: 51m1s, non-child: 36m18s, % non-child: 71.15%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 360.01M (360011594) - BuildTime: 2m40s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.00 GB (8590528921) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 5m44s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Buffer pool: - AllocTime: 3s965ms - CumulativeAllocationBytes: 357.66 GB (384038862848) - CumulativeAllocations: 174.95K (174948) - PeakReservation: 8.00 GB (8589934592) - PeakUnpinnedBytes: 169.02 GB (181479145472) - PeakUsedReservation: 8.00 GB (8589934592) - ReadIoBytes: 167.01 GB (179329564672) - ReadIoOps: 85.51K (85511) - ReadIoWaitTime: 25m52s - WriteIoBytes: 169.02 GB (181479145472) - WriteIoOps: 86.54K (86536) - WriteIoWaitTime: 4m32s Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled, Spilled Runtime filters: 0 of 4 Runtime Filters Published, 4 Disabled - BuildRowsPartitionTime: 2m37s - BuildRowsPartitioned: 360.01M (360011594) - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 14m54s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 48.12 KB (49280) - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) EXCHANGE_NODE (id=4):(Total: 8m29s, non-child: 7s931ms, % non-child: 1.56%) BytesReceived(16s000ms): 1.28 GB, 3.83 GB, 5.59 GB, 7.07 GB, 8.48 GB, 9.88 GB, 11.23 GB, 12.41 GB, 13.44 GB, 14.16 GB, 14.60 GB, 15.54 GB, 16.50 GB, 17.42 GB, 18.43 GB, 19.15 GB, 20.13 GB, 21.69 GB, 22.94 GB, 23.85 GB, 24.70 GB, 25.52 GB, 26.34 GB, 26.98 GB, 27.80 GB, 28.43 GB, 28.71 GB, 29.14 GB, 29.93 GB, 30.62 GB, 31.25 GB, 31.97 GB, 32.64 GB, 33.23 GB, 34.42 GB, 35.12 GB, 35.77 GB, 36.45 GB, 37.41 GB, 38.26 GB, 38.80 GB, 39.72 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 2s820ms - DeserializeRowBatchTimer: 1m17s - FirstBatchArrivalWaitTime: 225.592ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 707.29 K/sec - SendersBlockedTimer: 46s201ms - SendersBlockedTotalTimer(*): 46s201ms EXCHANGE_NODE (id=3):(Total: 6m15s, non-child: 11s158ms, % non-child: 2.97%) BytesReceived(1m4s): 9.99 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 631.63 MB, 3.74 GB, 7.24 GB, 10.11 GB, 12.66 GB, 15.79 GB, 18.96 GB, 23.65 GB, 29.15 GB, 34.26 GB, 38.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s421ms - DeserializeRowBatchTimer: 1m24s - FirstBatchArrivalWaitTime: 1s469ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 959.81 K/sec - SendersBlockedTimer: 14m16s - SendersBlockedTotalTimer(*): 14m16s Averaged Fragment F00:(Total: 22m1s, non-child: 14m6s, % non-child: 64.05%) split sizes: min: 12.56 GB, max: 12.56 GB, avg: 12.56 GB, stddev: 0 completion times: min:22m1s max:22m1s mean: 22m1s stddev:0.000ns execution rates: min:9.73 MB/sec max:9.73 MB/sec mean:9.73 MB/sec stddev:0.08 B/sec num instances: 1 - AverageThreadTokens: 5.53 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1201677440) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.66 GB (10376993786) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 14m20s - TotalStorageWaitTime: 16m40s - TotalThreadsInvoluntaryContextSwitches: 22.20K (22203) - TotalThreadsTotalWallClockTime: 2h1m Query (id=c24063c458ca1a05:8caca4e200000000): Summary: Session ID: f84755fd23448d06:ed727993551443af Session Type: BEESWAX Start Time: 2017-08-17 18:06:20.067066000 End Time: 2017-08-17 18:56:22.968335000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.10.0-SNAPSHOT RELEASE (build 473f717d0f38397f61f35e0ac9e32226d837a28a) User: tarmstrong Connected User: tarmstrong Delegated User: Network Address: ::ffff:127.0.0.1:55002 Default Db: tpch_60_parquet Sql Statement: select * from lineitem l1 join lineitem l2 on l1.l_orderkey = l2.l_orderkey and l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey and l1.l_linenumber = l2.l_linenumber where l1.l_tax < l2.l_tax Coordinator: tarmstrong-box2:22000 Query Options (non default): MEM_LIMIT=10737418240,MT_DOP=0 Plan: ---------------- Per-Host Resource Reservation: Memory=34.00MB Per-Host Resource Estimates: Memory=99.77GB F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=0B mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 05:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=0,1 row-size=534B cardinality=360011594 | F02:PLAN FRAGMENT [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] hosts=1 instances=1 Per-Host Resources: mem-estimate=98.39GB mem-reservation=34.00MB 02:HASH JOIN [INNER JOIN, PARTITIONED] | hash predicates: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | fk/pk conjuncts: l1.l_linenumber = l2.l_linenumber, l1.l_orderkey = l2.l_orderkey, l1.l_partkey = l2.l_partkey, l1.l_suppkey = l2.l_suppkey | other predicates: l1.l_tax < l2.l_tax | runtime filters: RF000 <- l2.l_linenumber, RF001 <- l2.l_orderkey, RF002 <- l2.l_partkey, RF003 <- l2.l_suppkey | mem-estimate=98.39GB mem-reservation=34.00MB spill-buffer=2.00MB | tuple-ids=0,1 row-size=534B cardinality=360011594 | |--04:EXCHANGE [HASH(l2.l_linenumber,l2.l_orderkey,l2.l_partkey,l2.l_suppkey)] | | mem-estimate=0B mem-reservation=0B | | tuple-ids=1 row-size=267B cardinality=360011594 | | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 | Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B | 01:SCAN HDFS [tpch_60_parquet.lineitem l2, RANDOM] | partitions=1/1 files=51 size=12.56GB | stats-rows=360011594 extrapolated-rows=disabled | table stats: rows=360011594 size=12.56GB | column stats: all | mem-estimate=704.00MB mem-reservation=0B | tuple-ids=1 row-size=267B cardinality=360011594 | 03:EXCHANGE [HASH(l1.l_linenumber,l1.l_orderkey,l1.l_partkey,l1.l_suppkey)] | mem-estimate=0B mem-reservation=0B | tuple-ids=0 row-size=267B cardinality=360011594 | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B 00:SCAN HDFS [tpch_60_parquet.lineitem l1, RANDOM] partitions=1/1 files=51 size=12.56GB runtime filters: RF000 -> l1.l_linenumber, RF001 -> l1.l_orderkey, RF002 -> l1.l_partkey, RF003 -> l1.l_suppkey stats-rows=360011594 extrapolated-rows=disabled table stats: rows=360011594 size=12.56GB column stats: all mem-estimate=704.00MB mem-reservation=0B tuple-ids=0 row-size=267B cardinality=360011594 ---------------- Estimated Per-Host Mem: 107126659482 Per-Host Memory Reservation: 35651584 Request Pool: default-pool Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail --------------------------------------------------------------------------------------------------------------------------- 05:EXCHANGE 1 3.890us 3.890us 0 360.01M 0 0 UNPARTITIONED 02:HASH JOIN 1 45m37s 45m37s 0 360.01M 8.00 GB 98.39 GB INNER JOIN, PARTITIONED |--04:EXCHANGE 1 18s648ms 18s648ms 360.01M 360.01M 0 0 HASH(l2.l_linenumber,l2.l_o... | 01:SCAN HDFS 1 2s109ms 2s109ms 360.01M 360.01M 1.12 GB 704.00 MB tpch_60_parquet.lineitem l2 03:EXCHANGE 1 14s968ms 14s968ms 360.01M 360.01M 0 0 HASH(l1.l_linenumber,l1.l_o... 00:SCAN HDFS 1 12s474ms 12s474ms 360.01M 360.01M 1.07 GB 704.00 MB tpch_60_parquet.lineitem l1 Errors: Planner Timeline: 24.673ms - Analysis finished: 2.883ms (2.883ms) - Equivalence classes computed: 5.305ms (2.421ms) - Single node plan created: 12.659ms (7.354ms) - Runtime filters computed: 13.034ms (375.049us) - Distributed plan created: 13.302ms (267.865us) - Planning finished: 24.673ms (11.370ms) Query Timeline: 50m2s - Query submitted: 15.651us (15.651us) - Planning finished: 27.016ms (27.001ms) - Submit for admission: 27.177ms (160.864us) - Completed admission: 27.207ms (29.332us) - Ready to start on 1 backends: 27.507ms (300.505us) - All 1 execution backends (4 fragment instances) started: 28.256ms (748.861us) - First dynamic filter received: 8m39s (8m39s) - Rows available: 50m2s (41m22s) - First row fetched: 50m2s (817.998ms) - Unregister query: 50m2s (3.324ms) - ComputeScanRangeAssignmentTimer: 59.423us ImpalaServer: - ClientFetchWaitTimer: 819.861ms - RowMaterializationTimer: 1.426ms Execution Profile c24063c458ca1a05:8caca4e200000000:(Total: 50m2s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 4 Filter routing table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 1 (1) N/A N/A true 2 2 0 REMOTE false 1 (1) N/A N/A true 1 2 0 REMOTE false 1 (1) N/A N/A true 0 2 0 REMOTE false 1 (1) N/A N/A true Backend startup latencies: Count: 1, min / max: 0 / 0, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 0, 99.9th %-ile: 0 Final filter table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- 3 2 0 REMOTE false 0 (1) 8m39s 8m39s false 2 2 0 REMOTE false 0 (1) 8m39s 8m39s false 1 2 0 REMOTE false 0 (1) 8m39s 8m39s false 0 2 0 REMOTE false 0 (1) 8m39s 8m39s false Per Node Peak Memory Usage: tarmstrong-box2:22000(9.64 GB) - FiltersReceived: 4 (4) - FinalizationTimer: 0.000ns Averaged Fragment F03:(Total: 50m2s, non-child: 921.705us, % non-child: 0.00%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:50m2s max:50m2s mean: 50m2s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.64 GB (10346222763) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 50m2s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 50m2s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 17.087us - ExecTreeExecTime: 2.966us - OpenTime: 50m2s - ExecTreeOpenTime: 50m2s - PrepareTime: 16.629ms - ExecTreePrepareTime: 10.848us PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 15.770ms, non-child: 15.770ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 13.561ms EXCHANGE_NODE (id=5):(Total: 50m2s, non-child: 50m2s, % non-child: 100.00%) - BytesReceived: 0 - ConvertRowBatchTime: 681.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 50m2s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Coordinator Fragment F03: Instance c24063c458ca1a05:8caca4e200000000 (host=tarmstrong-box2:22000):(Total: 50m2s, non-child: 921.705us, % non-child: 0.00%) MemoryUsage(1m4s): 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.00 KB (16384) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.64 GB (10346222763) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 50m2s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 50m2s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 0.000ns - TotalThreadsVoluntaryContextSwitches: 1 (1) Fragment Instance Lifecycle Timings: - ExecTime: 17.087us - ExecTreeExecTime: 2.966us - OpenTime: 50m2s - ExecTreeOpenTime: 50m2s - PrepareTime: 16.629ms - ExecTreePrepareTime: 10.848us PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 15.770ms, non-child: 15.770ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 13.561ms EXCHANGE_NODE (id=5):(Total: 50m2s, non-child: 3.890us, % non-child: 0.00%) BytesReceived(1m4s): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 0 - ConvertRowBatchTime: 681.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 50m2s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F02:(Total: 50m2s, non-child: 6s856ms, % non-child: 0.23%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:50m2s max:50m2s mean: 50m2s stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 1.18 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.11 GB (8706766649) - PeakReservation: 8.00 GB (8589934592) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.64 GB (10346222763) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 4m9s - TotalNetworkSendTime: 131.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 37.44K (37442) - TotalThreadsTotalWallClockTime: 58m49s - TotalThreadsSysTime: 22s664ms - TotalThreadsUserTime: 6m14s - TotalThreadsVoluntaryContextSwitches: 611.20K (611200) Fragment Instance Lifecycle Timings: - ExecTime: 41m14s - ExecTreeExecTime: 41m7s - OpenTime: 8m47s - ExecTreeOpenTime: 8m47s - PrepareTime: 23.851ms - ExecTreePrepareTime: 67.231us DataStreamSender (dst_id=5):(Total: 5.040us, non-child: 5.040us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 176.119ms, non-child: 176.119ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 49.525ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 110 (110) - NumInstructions: 5.01K (5009) - OptimizationTime: 106.036ms - PeakMemoryUsage: 2.45 MB (2564608) - PrepareTime: 18.429ms HASH_JOIN_NODE (id=2):(Total: 49m55s, non-child: 45m37s, % non-child: 91.42%) - BuildRows: 360.01M (360011594) - BuildTime: 7m4s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.00 GB (8590528958) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 8m17s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Buffer pool: - AllocTime: 2s582ms - CumulativeAllocationBytes: 356.91 GB (383231459328) - CumulativeAllocations: 174.56K (174563) - PeakReservation: 8.00 GB (8589934592) - PeakUnpinnedBytes: 169.02 GB (181479145472) - PeakUsedReservation: 8.00 GB (8589934592) - ReadIoBytes: 166.26 GB (178522161152) - ReadIoOps: 85.13K (85126) - ReadIoWaitTime: 28m19s - WriteIoBytes: 168.26 GB (180671741952) - WriteIoOps: 86.15K (86151) - WriteIoWaitTime: 11m3s Hash Join Builder (join_node_id=2): - BuildRowsPartitionTime: 6m56s - BuildRowsPartitioned: 360.01M (360011594) - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 15m56s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 48.12 KB (49280) - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) EXCHANGE_NODE (id=4):(Total: 1m34s, non-child: 1m34s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s809ms - DeserializeRowBatchTimer: 1m28s - FirstBatchArrivalWaitTime: 156.739ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 3.80 M/sec - SendersBlockedTimer: 4m5s - SendersBlockedTotalTimer(*): 4m5s EXCHANGE_NODE (id=3):(Total: 2m43s, non-child: 2m43s, % non-child: 100.00%) - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s943ms - DeserializeRowBatchTimer: 1m30s - FirstBatchArrivalWaitTime: 1s411ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.21 M/sec - SendersBlockedTimer: 13m52s - SendersBlockedTotalTimer(*): 13m52s Fragment F02: Instance c24063c458ca1a05:8caca4e200000003 (host=tarmstrong-box2:22000):(Total: 50m2s, non-child: 6s856ms, % non-child: 0.23%) MemoryUsage(1m4s): 6.80 GB, 8.10 GB, 8.10 GB, 8.10 GB, 8.10 GB, 8.10 GB, 8.10 GB, 8.10 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.08 GB, 8.07 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB, 8.06 GB ThreadUsage(1m4s): 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.18 - BloomFilterBytes: 64.00 MB (67108864) - PeakMemoryUsage: 8.11 GB (8706766649) - PeakReservation: 8.00 GB (8589934592) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.64 GB (10346222763) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 4m9s - TotalNetworkSendTime: 131.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 37.44K (37442) - TotalThreadsTotalWallClockTime: 58m49s - TotalThreadsSysTime: 22s664ms - TotalThreadsUserTime: 6m14s - TotalThreadsVoluntaryContextSwitches: 611.20K (611200) Fragment Instance Lifecycle Timings: - ExecTime: 41m14s - ExecTreeExecTime: 41m7s - OpenTime: 8m47s - ExecTreeOpenTime: 8m47s - PrepareTime: 23.851ms - ExecTreePrepareTime: 67.231us DataStreamSender (dst_id=5):(Total: 5.040us, non-child: 5.040us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 176.119ms, non-child: 176.119ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 49.525ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 110 (110) - NumInstructions: 5.01K (5009) - OptimizationTime: 106.036ms - PeakMemoryUsage: 2.45 MB (2564608) - PrepareTime: 18.429ms HASH_JOIN_NODE (id=2):(Total: 49m55s, non-child: 45m37s, % non-child: 91.42%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 360.01M (360011594) - BuildTime: 7m4s - NumHashTableBuildsSkipped: 0 (0) - PeakMemoryUsage: 8.00 GB (8590528958) - ProbeRows: 360.01M (360011594) - ProbeRowsPartitioned: 337.51M (337506624) - ProbeTime: 8m17s - RowsReturned: 0 (0) - RowsReturnedRate: 0 Buffer pool: - AllocTime: 2s582ms - CumulativeAllocationBytes: 356.91 GB (383231459328) - CumulativeAllocations: 174.56K (174563) - PeakReservation: 8.00 GB (8589934592) - PeakUnpinnedBytes: 169.02 GB (181479145472) - PeakUsedReservation: 8.00 GB (8589934592) - ReadIoBytes: 166.26 GB (178522161152) - ReadIoOps: 85.13K (85126) - ReadIoWaitTime: 28m19s - WriteIoBytes: 168.26 GB (180671741952) - WriteIoOps: 86.15K (86151) - WriteIoWaitTime: 11m3s Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled, Spilled Runtime filters: 0 of 4 Runtime Filters Published, 4 Disabled - BuildRowsPartitionTime: 6m56s - BuildRowsPartitioned: 360.01M (360011594) - HashBuckets: 1.07B (1073741824) - HashCollisions: 30.18M (30180602) - HashTablesBuildTime: 15m56s - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 48.12 KB (49280) - RepartitionTime: 0.000ns - SpilledPartitions: 15 (15) EXCHANGE_NODE (id=4):(Total: 1m34s, non-child: 18s648ms, % non-child: 19.69%) BytesReceived(16s000ms): 1.35 GB, 3.88 GB, 5.56 GB, 6.79 GB, 7.99 GB, 9.16 GB, 10.38 GB, 11.56 GB, 12.74 GB, 13.91 GB, 15.07 GB, 16.27 GB, 17.43 GB, 18.58 GB, 19.75 GB, 20.93 GB, 22.06 GB, 23.15 GB, 24.34 GB, 25.46 GB, 26.60 GB, 27.74 GB, 28.90 GB, 30.05 GB, 31.15 GB, 32.30 GB, 33.40 GB, 34.47 GB, 35.55 GB, 36.68 GB, 37.76 GB, 38.87 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s809ms - DeserializeRowBatchTimer: 1m28s - FirstBatchArrivalWaitTime: 156.739ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 3.80 M/sec - SendersBlockedTimer: 4m5s - SendersBlockedTotalTimer(*): 4m5s EXCHANGE_NODE (id=3):(Total: 2m43s, non-child: 14s968ms, % non-child: 9.17%) BytesReceived(1m4s): 9.99 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 10.23 MB, 1.39 GB, 5.90 GB, 10.44 GB, 14.85 GB, 19.38 GB, 23.45 GB, 27.62 GB, 32.12 GB, 36.37 GB, 39.64 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB, 39.92 GB - BytesReceived: 39.92 GB (42858904369) - ConvertRowBatchTime: 3s943ms - DeserializeRowBatchTimer: 1m30s - FirstBatchArrivalWaitTime: 1s411ms - PeakMemoryUsage: 0 - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 2.21 M/sec - SendersBlockedTimer: 13m52s - SendersBlockedTotalTimer(*): 13m52s Averaged Fragment F00:(Total: 18m31s, non-child: 13m42s, % non-child: 73.98%) split sizes: min: 12.56 GB, max: 12.56 GB, avg: 12.56 GB, stddev: 0 completion times: min:18m31s max:18m31s mean: 18m31s stddev:0.000ns execution rates: min:11.57 MB/sec max:11.57 MB/sec mean:11.57 MB/sec stddev:0.67 B/sec num instances: 1 - AverageThreadTokens: 5.47 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.07 GB (1150425070) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.64 GB (10346222763) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 13m59s - TotalStorageWaitTime: 3m7s - TotalThreadsInvoluntaryContextSwitches: 31.09K (31088) - TotalThreadsTotalWallClockTime: 1h41m - TotalThreadsSysTime: 2s500ms - TotalThreadsUserTime: 5m46s - TotalThreadsVoluntaryContextSwitches: 355.73K (355730) Fragment Instance Lifecycle Timings: - ExecTime: 18m31s - ExecTreeExecTime: 12s606ms - 1.07 GB, 1.07 GB, 1.07 GB, 1.07 GB, 1.04 GB, 690.05 MB, 672.34 MB, 574.52 MB, 555.78 MB, 679.97 MB, 655.93 MB, 548.64 MB, 635.56 MB, 728.82 MB, 521.22 MB, 574.98 MB, 608.17 MB, 637.16 MB, 665.00 MB, 667.71 MB, 647.40 MB, 642.67 MB ThreadUsage(32s000ms): 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5 - AverageThreadTokens: 5.47 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.07 GB (1150425070) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 9.64 GB (10346222763) - RowsProduced: 360.01M (360011594) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 13m59s - TotalStorageWaitTime: 3m7s - TotalThreadsInvoluntaryContextSwitches: 31.09K (31088) - TotalThreadsTotalWallClockTime: 1h41m - TotalThreadsSysTime: 2s500ms - TotalThreadsUserTime: 5m46s - TotalThreadsVoluntaryContextSwitches: 355.73K (355730) Fragment Instance Lifecycle Timings: - ExecTime: 18m31s - ExecTreeExecTime: 12s606ms - OpenTime: 46.569ms - ExecTreeOpenTime: 24.652us - PrepareTime: 17.971ms - ExecTreePrepareTime: 83.781us DataStreamSender (dst_id=3):(Total: 4m36s, non-child: 4m36s, % non-child: 100.00%) - BytesSent: 39.92 GB (42858904369) - NetworkThroughput(*): 2.18 GB/sec - OverallThroughput: 147.69 MB/sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 4m18s - TransmitDataRPCTime: 18s273ms - UncompressedRowBatchSize: 91.46 GB (98207171210) CodeGen:(Total: 63.058ms, non-child: 63.058ms, % non-child: 100.00%) - CodegenTime: 1.487ms - CompileTime: 7.701ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 36 (36) - NumInstructions: 655 (655) - OptimizationTime: 38.086ms - PeakMemoryUsage: 327.50 KB (335360) - PrepareTime: 15.400ms HDFS_SCAN_NODE (id=0):(Total: 12s474ms, non-child: 12s474ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 51 out of 51 Runtime filters: Not all filters arrived (arrived: [], missing [0, 1, 2, 3]), waited for 1s003ms Hdfs Read Thread Concurrency Bucket: 0:76.47% 1:6.298% 2:1.529% 3:3.419% 4:3.509% 5:2.654% 6:6.118% File Formats: PARQUET/SNAPPY:816 BytesRead(32s000ms): 900.64 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 989.95 MB, 996.83 MB, 1.11 GB, 1.91 GB, 2.56 GB, 3.21 GB, 4.05 GB, 4.74 GB, 5.30 GB, 6.09 GB, 6.94 GB, 7.31 GB, 8.01 GB, 8.68 GB, 9.44 GB, 10.18 GB, 10.86 GB, 11.50 GB, 12.15 GB - FooterProcessingTime: (Avg: 1s268ms ; Min: 1.399ms ; Max: 4s989ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 0.91 - AverageScannerThreadConcurrency: 4.47 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) : 8m39s - ExecTreeExecTime: 2s255ms - OpenTime: 20.754ms - ExecTreeOpenTime: 24.788us - PrepareTime: 17.248ms - ExecTreePrepareTime: 84.475us DataStreamSender (dst_id=4):(Total: 4m45s, non-child: 4m45s, % non-child: 100.00%) - BytesSent: 39.92 GB (42858904369) - NetworkThroughput(*): 2.12 GB/sec - OverallThroughput: 142.98 MB/sec - PeakMemoryUsage: 672.00 B (672) - RowsReturned: 360.01M (360011594) - SerializeBatchTime: 4m26s - TransmitDataRPCTime: 18s858ms - UncompressedRowBatchSize: 91.46 GB (98207171210) CodeGen:(Total: 36.898ms, non-child: 36.898ms, % non-child: 100.00%) - CodegenTime: 773.100us - CompileTime: 10.056ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2046588) - NumFunctions: 6 (6) - NumInstructions: 98 (98) - OptimizationTime: 9.895ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 14.344ms HDFS_SCAN_NODE (id=1):(Total: 2s109ms, non-child: 2s109ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.30 - AverageScannerThreadConcurrency: 2.13 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 22s213ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.12 GB (1199322115) - PerReadThreadRawHdfsThroughput: 83.01 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 1s261ms - RowBatchQueuePutWaitTime: 16m44s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 170.67 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 36.16K (36158) - ScannerThreadsTotalWallClockTime: 18m26s - MaterializeTupleTime(*): 56s797ms - ScannerThreadsSysTime: 1s520ms - ScannerThreadsUserTime: 1m25s - ScannerThreadsVoluntaryContextSwitches: 351.05K (351050) - TotalRawHdfsReadTime(*): 2m34s - TotalReadThroughput: 24.78 MB/sec Fragment F01: Instance c24063c458ca1a05:8caca4e200000002 (host=tarmstrong-box2:22000):(Total: 8m39s, non-child: 3m51s, % non-child: 44.52%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB MemoryUsage(16s000ms): 889.85 MB, 430.23 MB, 336.68 MB, 312.79 MB, 319.00 MB, 356.31 MB, 340.96 MB, 327.28 MB, 312.35 MB, 384.71 MB, 338.88 MB, 310.07 MB, 318.25 MB, 370.32 MB, 347.06 MB, 320.36 MB, 340.50 MB, 341.58 MB, 319.36 MB, 283.70 MB, 379.10 MB, 345.38 MB, 317.95 MB, 310.98 MB, 385.29 MB, 333.89 MB, 304.35 MB, 376.54 MB, 343.60 MB, 315.75 MB, 346.38 MB, 324.38 MB ThreadUsage(16s000ms): 6, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3 - AverageThreadTokens: 3.13 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.12 GB (1199557613) - PeakRS_SCAN_NODE (id=1):(Total: 2s109ms, non-child: 2s109ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:51/12.56 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 51 out of 51 Hdfs Read Thread Concurrency Bucket: 0:92% 1:2.987% 2:0.3854% 3:0.1927% 4:1.927% 5:0.4817% 6:2.023% File Formats: PARQUET/SNAPPY:816 BytesRead(16s000ms): 1.01 GB, 1.43 GB, 1.96 GB, 2.32 GB, 2.69 GB, 3.11 GB, 3.48 GB, 3.83 GB, 4.17 GB, 4.63 GB, 4.96 GB, 5.29 GB, 5.66 GB, 6.09 GB, 6.44 GB, 6.77 GB, 7.14 GB, 7.50 GB, 7.85 GB, 8.15 GB, 8.61 GB, 8.95 GB, 9.28 GB, 9.62 GB, 10.06 GB, 10.37 GB, 10.68 GB, 11.08 GB, 11.41 GB, 11.72 GB, 12.08 GB, 12.44 GB - FooterProcessingTime: (Avg: 39.657ms ; Min: 1.705ms ; Max: 157.228ms ; Number of samples: 51) - AverageHdfsReadThreadConcurrency: 0.30 - AverageScannerThreadConcurrency: 2.13 - BytesRead: 12.56 GB (13487696004) - BytesReadDataNodeCache: 0 - BytesReadLocal: 12.56 GB (13487696004) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 12.56 GB (13487696004) - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 867 (867) - DecompressionTime: 22s213ms - MaxCompressedTextFileLength: 0 - NumColumns: 16 (16) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 51 (51) - NumScannerThreadsStarted: 5 (5) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 1.12 GB (1199322115) - PerReadThreadRawHdfsThroughput: 83.01 MB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 1s261ms - RowBatchQueuePutWaitTime: 16m44s - RowsRead: 360.01M (360011594) - RowsReturned: 360.01M (360011594) - RowsReturnedRate: 170.67 M/sec - ScanRangesComplete: 51 (51) - ScannerThreadsInvoluntaryContextSwitches: 36.16K (36158) - ScannerThreadsTotalWallClockTime: 18m26s - MaterializeTupleTime(*): 56s797ms - ScannerThreadsSysTime: 1s520ms - ScannerThreadsUserTime: 1m25s - ScannerThreadsVoluntaryContextSwitches: 351.05K (351050) - TotalRawHdfsReadTime(*): 2m34s - TotalReadThroughput: 24.78 MB/sec