Query (id=d7448d78ac85ba6b:303b724200000000): Summary: Session ID: a34194af9e0f4619:66d7c62855acf8aa Session Type: BEESWAX Start Time: 2018-06-12 14:37:27.734635000 End Time: 2018-06-12 14:37:28.196937000 Query Type: QUERY Query State: EXCEPTION Query Status: Memory limit exceeded: Error occurred on backend vb0228.foo:22000 by fragment d7448d78ac85ba6b:303b72420000000c Memory left in process limit: 98.37 GB Memory left in query limit: -371.21 KB Query(d7448d78ac85ba6b:303b724200000000): memory limit exceeded. Limit=1.00 GB Reservation=816.00 MB ReservationLimit=819.20 MB OtherMemory=208.36 MB Total=1.00 GB Peak=1.00 GB Fragment d7448d78ac85ba6b:303b72420000000c: Reservation=360.00 MB OtherMemory=85.60 MB Total=445.60 MB Peak=445.60 MB HDFS_SCAN_NODE (id=2): Reservation=360.00 MB OtherMemory=85.45 MB Total=445.45 MB Peak=445.45 MB Exprs: Total=4.00 KB Peak=4.00 KB Queued Batches: Total=22.13 MB Peak=22.13 MB KrpcDataStreamSender (dst_id=13): Total=17.72 KB Peak=17.72 KB CodeGen: Total=1.68 KB Peak=679.50 KB Fragment d7448d78ac85ba6b:303b724200000023: Reservation=96.00 MB OtherMemory=43.09 MB Total=139.09 MB Peak=139.09 MB HDFS_SCAN_NODE (id=6): Reservation=96.00 MB OtherMemory=42.93 MB Total=138.93 MB Peak=138.93 MB Exprs: Total=4.00 KB Peak=4.00 KB Queued Batches: Total=26.10 MB Peak=26.10 MB KrpcDataStreamSender (dst_id=16): Total=17.72 KB Peak=17.72 KB CodeGen: Total=1.68 KB Peak=679.50 KB Fragment d7448d78ac85ba6b:303b72420000003a: Reservation=360.00 MB OtherMemory=79.67 MB Total=439.67 MB Peak=439.67 MB HDFS_SCAN_NODE (id=10): Reservation=360.00 MB OtherMemory=79.52 MB Total=439.52 MB Peak=439.52 MB Exprs: Total=4.00 KB Peak=4.00 KB Queued Batches: Total=22.20 MB Peak=22.20 MB KrpcDataStreamSender (dst_id=19): Total=17.72 KB Peak=17.72 KB CodeGen: Total=1.68 KB Peak=679.50 KB Impala Version: impalad version 2.13.0-SNAPSHOT RELEASE (build c6bb333720b515ede75e2b26668286fd7bcb53d4) User: mmokhtar Connected User: mmokhtar Delegated User: Network Address: 10.17.193.11:60138 Default Db: tpch_300_parquet Sql Statement: SELECT /* +straight_join */ count(*) FROM supplier JOIN /* +broadcast */ lineitem ON l_orderkey = s_nationkey WHERE l_partkey > 0 AND l_suppkey > 0 AND l_linenumber > 0 AND s_suppkey < 0 AND l_orderkey < 550000000 and l_comment > 'a' and l_shipdate > '1990' union all SELECT /* +straight_join */ count(*) FROM supplier JOIN /* +broadcast */ lineitem ON l_orderkey = s_nationkey WHERE l_partkey > 0 AND l_suppkey > 0 AND l_linenumber > 0 AND s_suppkey < 0 AND l_orderkey < 550000000 and l_comment > 'a' and l_shipdate > '1990' union all SELECT /* +straight_join */ count(*) FROM supplier JOIN /* +broadcast */ lineitem ON l_orderkey = s_nationkey WHERE l_partkey > 0 AND l_suppkey > 0 AND l_linenumber > 0 AND s_suppkey < 0 AND l_orderkey < 550000000 and l_comment > 'a' and l_shipdate > '1990' Coordinator: vb0206.foo:22000 Query Options (set by configuration): MEM_LIMIT=1073741824,RUNTIME_FILTER_MODE=0,MAX_ROW_SIZE=4194304 Query Options (set by configuration and planner): MEM_LIMIT=1073741824,RUNTIME_FILTER_MODE=0,MT_DOP=0,MAX_ROW_SIZE=4194304 Plan: ---------------- Max Per-Host Resource Reservation: Memory=198.00MB Threads=13 Per-Host Resource Estimates: Memory=57.42GB F09:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B thread-reservation=1 PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B thread-reservation=0 | 00:UNION | pass-through-operands: all | mem-estimate=0B mem-reservation=0B thread-reservation=0 | tuple-ids=9 row-size=8B cardinality=3 | |--21:AGGREGATE [FINALIZE] | | output: count:merge(*) | | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB thread-reservation=0 | | tuple-ids=8 row-size=8B cardinality=1 | | | 20:EXCHANGE [UNPARTITIONED] | | mem-estimate=0B mem-reservation=0B thread-reservation=0 | | tuple-ids=8 row-size=8B cardinality=1 | | | F06:PLAN FRAGMENT [RANDOM] hosts=9 instances=9 | Per-Host Resources: mem-estimate=18.62GB mem-reservation=42.00MB thread-reservation=2 | 12:AGGREGATE | | output: count(*) | | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB thread-reservation=0 | | tuple-ids=8 row-size=8B cardinality=1 | | | 11:HASH JOIN [INNER JOIN, BROADCAST] | | hash predicates: s_nationkey = l_orderkey | | fk/pk conjuncts: none | | mem-estimate=18.53GB mem-reservation=38.00MB spill-buffer=2.00MB thread-reservation=0 | | tuple-ids=6,7 row-size=116B cardinality=1233124 | | | |--19:EXCHANGE [BROADCAST] | | | mem-estimate=0B mem-reservation=0B thread-reservation=0 | | | tuple-ids=7 row-size=100B cardinality=179998909 | | | | | F07:PLAN FRAGMENT [RANDOM] hosts=14 instances=14 | | Per-Host Resources: mem-estimate=528.00MB mem-reservation=24.00MB thread-reservation=2 | | 10:SCAN HDFS [tpch_300_parquet.lineitem, RANDOM] | | partitions=1/1 files=264 size=64.36GB | | predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | | stored statistics: | | table: rows=1799989091 size=64.36GB | | columns: all | | extrapolated-rows=1799989091 max-scan-range-rows=6921690 | | parquet statistics predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | | parquet dictionary predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | | mem-estimate=528.00MB mem-reservation=24.00MB thread-reservation=1 | | tuple-ids=7 row-size=100B cardinality=179998909 | | | 09:SCAN HDFS [tpch_300_parquet.supplier, RANDOM] | partitions=1/1 files=9 size=233.61MB | predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 | stored statistics: | table: rows=3000000 size=233.61MB | columns: all | extrapolated-rows=3000000 max-scan-range-rows=339922 | parquet statistics predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 | parquet dictionary predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 | mem-estimate=80.00MB mem-reservation=4.00MB thread-reservation=1 | tuple-ids=6 row-size=16B cardinality=300000 | |--18:AGGREGATE [FINALIZE] | | output: count:merge(*) | | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB thread-reservation=0 | | tuple-ids=5 row-size=8B cardinality=1 | | | 17:EXCHANGE [UNPARTITIONED] | | mem-estimate=0B mem-reservation=0B thread-reservation=0 | | tuple-ids=5 row-size=8B cardinality=1 | | | F03:PLAN FRAGMENT [RANDOM] hosts=9 instances=9 | Per-Host Resources: mem-estimate=18.62GB mem-reservation=42.00MB thread-reservation=2 | 08:AGGREGATE | | output: count(*) | | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB thread-reservation=0 | | tuple-ids=5 row-size=8B cardinality=1 | | | 07:HASH JOIN [INNER JOIN, BROADCAST] | | hash predicates: s_nationkey = l_orderkey | | fk/pk conjuncts: none | | mem-estimate=18.53GB mem-reservation=38.00MB spill-buffer=2.00MB thread-reservation=0 | | tuple-ids=3,4 row-size=116B cardinality=1233124 | | | |--16:EXCHANGE [BROADCAST] | | | mem-estimate=0B mem-reservation=0B thread-reservation=0 | | | tuple-ids=4 row-size=100B cardinality=179998909 | | | | | F04:PLAN FRAGMENT [RANDOM] hosts=14 instances=14 | | Per-Host Resources: mem-estimate=528.00MB mem-reservation=24.00MB thread-reservation=2 | | 06:SCAN HDFS [tpch_300_parquet.lineitem, RANDOM] | | partitions=1/1 files=264 size=64.36GB | | predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | | stored statistics: | | table: rows=1799989091 size=64.36GB | | columns: all | | extrapolated-rows=1799989091 max-scan-range-rows=6921690 | | parquet statistics predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | | parquet dictionary predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | | mem-estimate=528.00MB mem-reservation=24.00MB thread-reservation=1 | | tuple-ids=4 row-size=100B cardinality=179998909 | | | 05:SCAN HDFS [tpch_300_parquet.supplier, RANDOM] | partitions=1/1 files=9 size=233.61MB | predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 | stored statistics: | table: rows=3000000 size=233.61MB | columns: all | extrapolated-rows=3000000 max-scan-range-rows=339922 | parquet statistics predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 | parquet dictionary predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 | mem-estimate=80.00MB mem-reservation=4.00MB thread-reservation=1 | tuple-ids=3 row-size=16B cardinality=300000 | 15:AGGREGATE [FINALIZE] | output: count:merge(*) | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB thread-reservation=0 | tuple-ids=2 row-size=8B cardinality=1 | 14:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B thread-reservation=0 | tuple-ids=2 row-size=8B cardinality=1 | F00:PLAN FRAGMENT [RANDOM] hosts=9 instances=9 Per-Host Resources: mem-estimate=18.62GB mem-reservation=42.00MB thread-reservation=2 04:AGGREGATE | output: count(*) | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB thread-reservation=0 | tuple-ids=2 row-size=8B cardinality=1 | 03:HASH JOIN [INNER JOIN, BROADCAST] | hash predicates: s_nationkey = l_orderkey | fk/pk conjuncts: none | mem-estimate=18.53GB mem-reservation=38.00MB spill-buffer=2.00MB thread-reservation=0 | tuple-ids=0,1 row-size=116B cardinality=1233124 | |--13:EXCHANGE [BROADCAST] | | mem-estimate=0B mem-reservation=0B thread-reservation=0 | | tuple-ids=1 row-size=100B cardinality=179998909 | | | F01:PLAN FRAGMENT [RANDOM] hosts=14 instances=14 | Per-Host Resources: mem-estimate=528.00MB mem-reservation=24.00MB thread-reservation=2 | 02:SCAN HDFS [tpch_300_parquet.lineitem, RANDOM] | partitions=1/1 files=264 size=64.36GB | predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | stored statistics: | table: rows=1799989091 size=64.36GB | columns: all | extrapolated-rows=1799989091 max-scan-range-rows=6921690 | parquet statistics predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | parquet dictionary predicates: l_linenumber > 0, l_orderkey < 550000000, l_partkey > 0, l_suppkey > 0, l_shipdate > '1990', l_comment > 'a' | mem-estimate=528.00MB mem-reservation=24.00MB thread-reservation=1 | tuple-ids=1 row-size=100B cardinality=179998909 | 01:SCAN HDFS [tpch_300_parquet.supplier, RANDOM] partitions=1/1 files=9 size=233.61MB predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 stored statistics: table: rows=3000000 size=233.61MB columns: all extrapolated-rows=3000000 max-scan-range-rows=339922 parquet statistics predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 parquet dictionary predicates: s_suppkey < 0, tpch_300_parquet.supplier.s_nationkey < 550000000 mem-estimate=80.00MB mem-reservation=4.00MB thread-reservation=1 tuple-ids=0 row-size=16B cardinality=300000 ---------------- Estimated Per-Host Mem: 61650952192 Per Host Min Memory Reservation: vb0204.foo:22000(72.00 MB) vb0206.foo:22000(198.00 MB) vb0208.foo:22000(198.00 MB) vb0212.foo:22000(72.00 MB) vb0214.foo:22000(198.00 MB) vb0216.foo:22000(198.00 MB) vb0218.foo:22000(198.00 MB) vb0220.foo:22000(72.00 MB) vb0222.foo:22000(198.00 MB) vb0224.foo:22000(198.00 MB) vb0228.foo:22000(72.00 MB) vb0230.foo:22000(198.00 MB) vb0232.foo:22000(198.00 MB) vb0234.foo:22000(72.00 MB) Request Pool: root.default Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail ------------------------------------------------------------------------------------------------------------------------ 00:UNION 1 0.000ns 0.000ns 0 3 0 0 |--18:AGGREGATE 1 0.000ns 0.000ns 0 1 0 10.00 MB FINALIZE | 17:EXCHANGE 1 0.000ns 0.000ns 0 1 0 0 UNPARTITIONED | 08:AGGREGATE 9 0.000ns 0.000ns 0 1 0 10.00 MB | 07:HASH JOIN 9 0.000ns 0.000ns 0 1.23M 0 18.53 GB INNER JOIN, BROADCAST | |--16:EXCHANGE 9 0.000ns 0.000ns 0 180.00M 0 0 BROADCAST | | 06:SCAN HDFS 14 0.000ns 0.000ns 0 180.00M 0 528.00 MB tpch_300_parquet.lineitem | 05:SCAN HDFS 9 0.000ns 0.000ns 0 300.00K 0 80.00 MB tpch_300_parquet.supplier |--21:AGGREGATE 1 0.000ns 0.000ns 0 1 0 10.00 MB FINALIZE | 20:EXCHANGE 1 0.000ns 0.000ns 0 1 0 0 UNPARTITIONED | 12:AGGREGATE 9 0.000ns 0.000ns 0 1 0 10.00 MB | 11:HASH JOIN 9 0.000ns 0.000ns 0 1.23M 0 18.53 GB INNER JOIN, BROADCAST | |--19:EXCHANGE 9 0.000ns 0.000ns 0 180.00M 0 0 BROADCAST | | 10:SCAN HDFS 14 0.000ns 0.000ns 0 180.00M 0 528.00 MB tpch_300_parquet.lineitem | 09:SCAN HDFS 9 0.000ns 0.000ns 0 300.00K 0 80.00 MB tpch_300_parquet.supplier 15:AGGREGATE 1 0.000ns 0.000ns 0 1 0 10.00 MB FINALIZE 14:EXCHANGE 1 0.000ns 0.000ns 0 1 0 0 UNPARTITIONED 04:AGGREGATE 9 0.000ns 0.000ns 0 1 0 10.00 MB 03:HASH JOIN 9 0.000ns 0.000ns 0 1.23M 0 18.53 GB INNER JOIN, BROADCAST |--13:EXCHANGE 9 0.000ns 0.000ns 0 180.00M 0 0 BROADCAST | 02:SCAN HDFS 14 5.149ms 72.091ms 2.05K 180.00M 445.45 MB 528.00 MB tpch_300_parquet.lineitem 01:SCAN HDFS 9 0.000ns 0.000ns 0 300.00K 0 80.00 MB tpch_300_parquet.supplier Errors: Memory limit exceeded: Error occurred on backend vb0228.foo:22000 by fragment d7448d78ac85ba6b:303b72420000000c Memory left in process limit: 98.37 GB Memory left in query limit: -371.21 KB Query(d7448d78ac85ba6b:303b724200000000): memory limit exceeded. Limit=1.00 GB Reservation=816.00 MB ReservationLimit=819.20 MB OtherMemory=208.36 MB Total=1.00 GB Peak=1.00 GB Fragment d7448d78ac85ba6b:303b72420000000c: Reservation=360.00 MB OtherMemory=85.60 MB Total=445.60 MB Peak=445.60 MB HDFS_SCAN_NODE (id=2): Reservation=360.00 MB OtherMemory=85.45 MB Total=445.45 MB Peak=445.45 MB Exprs: Total=4.00 KB Peak=4.00 KB Queued Batches: Total=22.13 MB Peak=22.13 MB KrpcDataStreamSender (dst_id=13): Total=17.72 KB Peak=17.72 KB CodeGen: Total=1.68 KB Peak=679.50 KB Fragment d7448d78ac85ba6b:303b724200000023: Reservation=96.00 MB OtherMemory=43.09 MB Total=139.09 MB Peak=139.09 MB HDFS_SCAN_NODE (id=6): Reservation=96.00 MB OtherMemory=42.93 MB Total=138.93 MB Peak=138.93 MB Exprs: Total=4.00 KB Peak=4.00 KB Queued Batches: Total=26.10 MB Peak=26.10 MB KrpcDataStreamSender (dst_id=16): Total=17.72 KB Peak=17.72 KB CodeGen: Total=1.68 KB Peak=679.50 KB Fragment d7448d78ac85ba6b:303b72420000003a: Reservation=360.00 MB OtherMemory=79.67 MB Total=439.67 MB Peak=439.67 MB HDFS_SCAN_NODE (id=10): Reservation=360.00 MB OtherMemory=79.52 MB Total=439.52 MB Peak=439.52 MB Exprs: Total=4.00 KB Peak=4.00 KB Queued Batches: Total=22.20 MB Peak=22.20 MB KrpcDataStreamSender (dst_id=19): Total=17.72 KB Peak=17.72 KB CodeGen: Total=1.68 KB Peak=679.50 KB (1 of 2 similar) Query Compilation: 35.486ms - Metadata of all 2 tables cached: 3.043ms (3.043ms) - Analysis finished: 9.163ms (6.120ms) - Value transfer graph computed: 9.386ms (222.889us) - Single node plan created: 25.813ms (16.426ms) - Distributed plan created: 26.555ms (742.129us) - Lineage info computed: 26.737ms (182.036us) - Planning finished: 35.486ms (8.749ms) Query Timeline: 467.511ms - Query submitted: 75.185us (75.185us) - Planning finished: 42.206ms (42.131ms) - Submit for admission: 45.211ms (3.005ms) - Completed admission: 45.293ms (82.070us) - Ready to start on 14 backends: 45.863ms (569.909us) - All 14 execution backends (70 fragment instances) started: 52.634ms (6.771ms) - Execution error: 425.645ms (373.010ms) - Rows available: 426.483ms (838.166us) - Released admission control resources: 433.452ms (6.969ms) - Unregister query: 462.318ms (28.865ms) - ComputeScanRangeAssignmentTimer: 1.652ms ImpalaServer: - ClientFetchWaitTimer: 0.000ns - RowMaterializationTimer: 0.000ns Execution Profile d7448d78ac85ba6b:303b724200000000:(Total: 380.622ms, non-child: 0.000ns, % non-child: 0.00%) Backend startup latencies: Count: 14, min / max: 2ms / 3ms, 25th %-ile: 2ms, 50th %-ile: 2ms, 75th %-ile: 3ms, 90th %-ile: 3ms, 95th %-ile: 3ms, 99.9th %-ile: 3ms Slowest Backend to startup: vb0214.foo:22000 Per Node Peak Memory Usage: vb0204.foo:22000(0) vb0206.foo:22000(0) vb0208.foo:22000(0) vb0212.foo:22000(0) vb0214.foo:22000(0) vb0216.foo:22000(0) vb0218.foo:22000(0) vb0220.foo:22000(0) vb0222.foo:22000(0) vb0224.foo:22000(0) vb0228.foo:22000(1.00 GB) vb0230.foo:22000(0) vb0232.foo:22000(0) vb0234.foo:22000(0) - FiltersReceived: 0 (0) - FinalizationTimer: 0.000ns - NumBackends: 14 (14) - NumFragmentInstances: 70 (70) - NumFragments: 7 (7) Averaged Fragment F09: split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:384.350ms max:384.350ms mean: 384.350ms stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 Coordinator Fragment F09: Instance d7448d78ac85ba6b:303b724200000000 (host=vb0206.foo:22000): Averaged Fragment F00: split sizes: min: 25.70 MB, max: 26.47 MB, avg: 25.96 MB, stddev: 363.28 KB completion times: min:381.839ms max:384.469ms mean: 383.829ms stddev:742.793us execution rates: min:66.85 MB/sec max:69.32 MB/sec mean:67.63 MB/sec stddev:1.01 MB/sec num instances: 9 Fragment F00: Instance d7448d78ac85ba6b:303b724200000011 (host=vb0206.foo:22000): Instance d7448d78ac85ba6b:303b724200000015 (host=vb0208.foo:22000): Instance d7448d78ac85ba6b:303b724200000016 (host=vb0214.foo:22000): Instance d7448d78ac85ba6b:303b72420000000f (host=vb0216.foo:22000): Instance d7448d78ac85ba6b:303b724200000012 (host=vb0218.foo:22000): Instance d7448d78ac85ba6b:303b724200000010 (host=vb0222.foo:22000): Instance d7448d78ac85ba6b:303b724200000017 (host=vb0224.foo:22000): Instance d7448d78ac85ba6b:303b724200000014 (host=vb0230.foo:22000): Instance d7448d78ac85ba6b:303b724200000013 (host=vb0232.foo:22000): Averaged Fragment F01:(Total: 26.720ms, non-child: 0.000ns, % non-child: 0.00%) split sizes: min: 4.21 GB, max: 4.71 GB, avg: 4.60 GB, stddev: 147.33 MB completion times: min:375.818ms max:384.903ms mean: 383.323ms stddev:2.228ms execution rates: min:10.93 GB/sec max:12.51 GB/sec mean:11.99 GB/sec stddev:414.71 MB/sec num instances: 14 - AverageThreadTokens: 1.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 445.60 MB (467245594) - PeakReservation: 360.00 MB (377487360) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 1.00 GB (1074121945) - RowsProduced: 2.05K (2048) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 185.670ms - TotalStorageWaitTime: 1s270ms - TotalThreadsInvoluntaryContextSwitches: 13 (13) - TotalThreadsTotalWallClockTime: 372.826ms - TotalThreadsSysTime: 2.999ms - TotalThreadsUserTime: 112.983ms - TotalThreadsVoluntaryContextSwitches: 40 (40) Buffer pool: - AllocTime: 0.000ns - CumulativeAllocationBytes: 0 - CumulativeAllocations: 0 (0) - PeakReservation: 0 - PeakUnpinnedBytes: 0 - PeakUsedReservation: 0 - ReadIoBytes: 0 - ReadIoOps: 0 (0) - ReadIoWaitTime: 0.000ns - ReservationLimit: 0 - WriteIoBytes: 0 - WriteIoOps: 0 (0) - WriteIoWaitTime: 0.000ns Fragment Instance Lifecycle Timings: - ExecTime: 261.114ms - ExecTreeExecTime: 71.028ms - OpenTime: 111.717ms - ExecTreeOpenTime: 79.642us - PrepareTime: 1.183ms - ExecTreePrepareTime: 470.147us KrpcDataStreamSender (dst_id=13):(Total: 190.199ms, non-child: 190.199ms, % non-child: 100.00%) - EosSent: 0 (0) - NetworkThroughput: 38.62 MB/sec - PeakMemoryUsage: 17.72 KB (18144) - RowsSent: 2.05K (2048) - RpcFailure: 0 (0) - RpcRetry: 0 (0) - SerializeBatchTime: 3.621ms - TotalBytesSent: 745.40 KB (763290) - UncompressedRowBatchSize: 1.92 MB (2016639) HDFS_SCAN_NODE (id=2):(Total: 72.091ms, non-child: 72.091ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 0.00 - BytesRead: 361.46 MB (379023360) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 0 (0) - CollectionItemsRead: 0 (0) - DecompressionTime: 141.923ms - MaxCompressedTextFileLength: 0 - NumColumns: 6 (6) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 0 (0) - NumRowGroups: 15 (15) - NumScannerThreadReservationsDenied: 1 (1) - NumScannerThreadsStarted: 15 (15) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 445.45 MB (467084421) - PerReadThreadRawHdfsThroughput: 688.27 MB/sec - RemoteScanRanges: 0 (0) - RowBatchBytesEnqueued: 21.32 MB (22360164) - RowBatchQueueCapacity: 0 (0) - RowBatchQueueGetWaitTime: 0.000ns - RowBatchQueuePeakMemoryUsage: 0 - RowBatchQueuePutWaitTime: 0.000ns - RowBatchesEnqueued: 137 (137) - RowsRead: 579.58K (579584) - RowsReturned: 2.05K (2048) - RowsReturnedRate: 28.41 K/sec - ScanRangesComplete: 0 (0) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 0.000ns - MaterializeTupleTime(*): 0.000ns - ScannerThreadsSysTime: 0.000ns - ScannerThreadsUserTime: 0.000ns - ScannerThreadsVoluntaryContextSwitches: 0 (0) - TotalRawHdfsOpenFileTime(*): 1.484ms - TotalRawHdfsReadTime(*): 525.179ms - TotalReadThroughput: 0.00 /sec Buffer pool: - AllocTime: 11.630ms - CumulativeAllocationBytes: 361.88 MB (379453440) - CumulativeAllocations: 105 (105) - PeakReservation: 360.00 MB (377487360) - PeakUnpinnedBytes: 0 - PeakUsedReservation: 360.00 MB (377487360) - ReadIoBytes: 0 - ReadIoOps: 0 (0) - ReadIoWaitTime: 0.000ns - WriteIoBytes: 0 - WriteIoOps: 0 (0) - WriteIoWaitTime: 0.000ns CodeGen:(Total: 111.531ms, non-child: 111.531ms, % non-child: 100.00%) - CodegenInvoluntaryContextSwitches: 12 (12) - CodegenTotalWallClockTime: 111.525ms - CodegenSysTime: 0.000ns - CodegenUserTime: 110.983ms - CodegenVoluntaryContextSwitches: 2 (2) - CompileTime: 27.788ms - IrGenerationTime: 3.640ms - LoadTime: 0.000ns - ModuleBitcodeSize: 2.26 MB (2371828) - NumFunctions: 57 (57) - NumInstructions: 1.36K (1359) - OptimizationTime: 59.906ms - PeakMemoryUsage: 679.50 KB (695808) - PrepareTime: 19.389ms Fragment F01: Instance d7448d78ac85ba6b:303b72420000000c (host=vb0228.foo:22000):(Total: 374.084ms, non-child: 261.567us, % non-child: 0.07%) Hdfs split stats (:<# splits>/): 5:1/253.40 MB 10:2/506.78 MB 3:3/760.19 MB 1:2/506.79 MB 0:1/253.38 MB 9:2/506.79 MB 4:2/506.77 MB 2:1/253.39 MB 8:1/253.41 MB 6:3/760.22 MB 7:1/253.40 MB Fragment Instance Lifecycle Event Timeline: 374.361ms - Prepare Finished: 1.529ms (1.529ms) - Open Finished: 113.248ms (111.719ms) - First Batch Produced: 184.272ms (71.023ms) - First Batch Sent: 185.451ms (1.179ms) - ExecInternal Finished: 374.361ms (188.910ms) MemoryUsage(500.000ms): 705.22 KB ThreadUsage(500.000ms): 1 - AverageThreadTokens: 1.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 445.60 MB (467245594) - PeakReservation: 360.00 MB (377487360) - PeakUsedReservation: 0 - PerHostPeakMemUsage: 1.00 GB (1074121945) - RowsProduced: 2.05K (2048) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 185.670ms - TotalStorageWaitTime: 1s270ms - TotalThreadsInvoluntaryContextSwitches: 13 (13) - TotalThreadsTotalWallClockTime: 372.826ms - TotalThreadsSysTime: 2.999ms - TotalThreadsUserTime: 112.983ms - TotalThreadsVoluntaryContextSwitches: 40 (40) Buffer pool: - AllocTime: 0.000ns - CumulativeAllocationBytes: 0 - CumulativeAllocations: 0 (0) - PeakReservation: 0 - PeakUnpinnedBytes: 0 - PeakUsedReservation: 0 - ReadIoBytes: 0 - ReadIoOps: 0 (0) - ReadIoWaitTime: 0.000ns - ReservationLimit: 0 - WriteIoBytes: 0 - WriteIoOps: 0 (0) - WriteIoWaitTime: 0.000ns Fragment Instance Lifecycle Timings: - ExecTime: 261.114ms - ExecTreeExecTime: 71.028ms - OpenTime: 111.717ms - ExecTreeOpenTime: 79.642us - PrepareTime: 1.183ms - ExecTreePrepareTime: 470.147us KrpcDataStreamSender (dst_id=13):(Total: 190.199ms, non-child: 4.523ms, % non-child: 2.38%) BytesSent(500.000ms): 0 - EosSent: 0 (0) - NetworkThroughput: 38.62 MB/sec - PeakMemoryUsage: 17.72 KB (18144) - RowsSent: 2.05K (2048) - RpcFailure: 0 (0) - RpcRetry: 0 (0) - SerializeBatchTime: 3.621ms - TotalBytesSent: 745.40 KB (763290) - UncompressedRowBatchSize: 1.92 MB (2016639) HDFS_SCAN_NODE (id=2):(Total: 72.091ms, non-child: 72.091ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 5:1/253.40 MB 10:2/506.78 MB 3:3/760.19 MB 1:2/506.79 MB 0:1/253.38 MB 9:2/506.79 MB 4:2/506.77 MB 2:1/253.39 MB 8:1/253.41 MB 6:3/760.22 MB 7:1/253.40 MB ExecOption: PARQUET Codegen Enabled BytesRead(500.000ms): 0 - FooterProcessingTime: (Avg: 1.252ms ; Min: 581.435us ; Max: 2.586ms ; Number of samples: 15) - InitialRangeActualReservation: (Avg: 24.00 MB (25165824) ; Min: 24.00 MB (25165824) ; Max: 24.00 MB (25165824) ; Number of samples: 15) - InitialRangeIdealReservation: (Avg: 128.00 KB (131072) ; Min: 128.00 KB (131072) ; Max: 128.00 KB (131072) ; Number of samples: 15) - ParquetRowGroupActualReservation: (Avg: 24.00 MB (25165824) ; Min: 24.00 MB (25165824) ; Max: 24.00 MB (25165824) ; Number of samples: 15) - ParquetRowGroupIdealReservation: (Avg: 106.00 MB (111149056) ; Min: 106.00 MB (111149056) ; Max: 106.00 MB (111149056) ; Number of samples: 15) - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 0.00 - BytesRead: 361.46 MB (379023360) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 0 (0) - CollectionItemsRead: 0 (0) - DecompressionTime: 141.923ms - MaxCompressedTextFileLength: 0 - NumColumns: 6 (6) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 0 (0) - NumRowGroups: 15 (15) - NumScannerThreadReservationsDenied: 1 (1) - NumScannerThreadsStarted: 15 (15) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 445.45 MB (467084421) - PerReadThreadRawHdfsThroughput: 688.27 MB/sec - RemoteScanRanges: 0 (0) - RowBatchBytesEnqueued: 21.32 MB (22360164) - RowBatchQueueCapacity: 0 (0) - RowBatchQueueGetWaitTime: 0.000ns - RowBatchQueuePeakMemoryUsage: 0 - RowBatchQueuePutWaitTime: 0.000ns - RowBatchesEnqueued: 137 (137) - RowsRead: 579.58K (579584) - RowsReturned: 2.05K (2048) - RowsReturnedRate: 28.41 K/sec - ScanRangesComplete: 0 (0) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 0.000ns - MaterializeTupleTime(*): 0.000ns - ScannerThreadsSysTime: 0.000ns - ScannerThreadsUserTime: 0.000ns - ScannerThreadsVoluntaryContextSwitches: 0 (0) - TotalRawHdfsOpenFileTime(*): 1.484ms - TotalRawHdfsReadTime(*): 525.179ms - TotalReadThroughput: 0.00 /sec Buffer pool: - AllocTime: 11.630ms - CumulativeAllocationBytes: 361.88 MB (379453440) - CumulativeAllocations: 105 (105) - PeakReservation: 360.00 MB (377487360) - PeakUnpinnedBytes: 0 - PeakUsedReservation: 360.00 MB (377487360) - ReadIoBytes: 0 - ReadIoOps: 0 (0) - ReadIoWaitTime: 0.000ns - WriteIoBytes: 0 - WriteIoOps: 0 (0) - WriteIoWaitTime: 0.000ns CodeGen:(Total: 111.531ms, non-child: 111.531ms, % non-child: 100.00%) - CodegenInvoluntaryContextSwitches: 12 (12) - CodegenTotalWallClockTime: 111.525ms - CodegenSysTime: 0.000ns - CodegenUserTime: 110.983ms - CodegenVoluntaryContextSwitches: 2 (2) - CompileTime: 27.788ms - IrGenerationTime: 3.640ms - LoadTime: 0.000ns - ModuleBitcodeSize: 2.26 MB (2371828) - NumFunctions: 57 (57) - NumInstructions: 1.36K (1359) - OptimizationTime: 59.906ms - PeakMemoryUsage: 679.50 KB (695808) - PrepareTime: 19.389ms Instance d7448d78ac85ba6b:303b72420000000e (host=vb0204.foo:22000): Instance d7448d78ac85ba6b:303b724200000006 (host=vb0206.foo:22000): Instance d7448d78ac85ba6b:303b724200000007 (host=vb0208.foo:22000): Instance d7448d78ac85ba6b:303b724200000003 (host=vb0212.foo:22000): Instance d7448d78ac85ba6b:303b724200000004 (host=vb0214.foo:22000): Instance d7448d78ac85ba6b:303b724200000002 (host=vb0216.foo:22000): Instance d7448d78ac85ba6b:303b72420000000d (host=vb0218.foo:22000): Instance d7448d78ac85ba6b:303b72420000000b (host=vb0220.foo:22000): Instance d7448d78ac85ba6b:303b72420000000a (host=vb0222.foo:22000): Instance d7448d78ac85ba6b:303b724200000005 (host=vb0224.foo:22000): Instance d7448d78ac85ba6b:303b724200000001 (host=vb0230.foo:22000): Instance d7448d78ac85ba6b:303b724200000009 (host=vb0232.foo:22000): Instance d7448d78ac85ba6b:303b724200000008 (host=vb0234.foo:22000): Averaged Fragment F03: split sizes: min: 25.70 MB, max: 26.47 MB, avg: 25.96 MB, stddev: 363.28 KB completion times: min:381.827ms max:384.462ms mean: 383.822ms stddev:744.870us execution rates: min:66.85 MB/sec max:69.32 MB/sec mean:67.63 MB/sec stddev:1.01 MB/sec num instances: 9 Fragment F03: Instance d7448d78ac85ba6b:303b724200000028 (host=vb0206.foo:22000): Instance d7448d78ac85ba6b:303b72420000002c (host=vb0208.foo:22000): Instance d7448d78ac85ba6b:303b72420000002d (host=vb0214.foo:22000): Instance d7448d78ac85ba6b:303b724200000026 (host=vb0216.foo:22000): Instance d7448d78ac85ba6b:303b724200000029 (host=vb0218.foo:22000): Instance d7448d78ac85ba6b:303b724200000027 (host=vb0222.foo:22000): Instance d7448d78ac85ba6b:303b72420000002e (host=vb0224.foo:22000): Instance d7448d78ac85ba6b:303b72420000002b (host=vb0230.foo:22000): Instance d7448d78ac85ba6b:303b72420000002a (host=vb0232.foo:22000): Averaged Fragment F04: split sizes: min: 4.21 GB, max: 4.71 GB, avg: 4.60 GB, stddev: 147.33 MB completion times: min:381.824ms max:384.901ms mean: 383.917ms stddev:798.091us execution rates: min:10.93 GB/sec max:12.31 GB/sec mean:11.97 GB/sec stddev:395.04 MB/sec num instances: 14 Fragment F04: Instance d7448d78ac85ba6b:303b724200000025 (host=vb0204.foo:22000): Instance d7448d78ac85ba6b:303b72420000001d (host=vb0206.foo:22000): Instance d7448d78ac85ba6b:303b72420000001e (host=vb0208.foo:22000): Instance d7448d78ac85ba6b:303b72420000001a (host=vb0212.foo:22000): Instance d7448d78ac85ba6b:303b72420000001b (host=vb0214.foo:22000): Instance d7448d78ac85ba6b:303b724200000019 (host=vb0216.foo:22000): Instance d7448d78ac85ba6b:303b724200000024 (host=vb0218.foo:22000): Instance d7448d78ac85ba6b:303b724200000022 (host=vb0220.foo:22000): Instance d7448d78ac85ba6b:303b724200000021 (host=vb0222.foo:22000): Instance d7448d78ac85ba6b:303b72420000001c (host=vb0224.foo:22000): Instance d7448d78ac85ba6b:303b724200000023 (host=vb0228.foo:22000): Instance d7448d78ac85ba6b:303b724200000018 (host=vb0230.foo:22000): Instance d7448d78ac85ba6b:303b724200000020 (host=vb0232.foo:22000): Instance d7448d78ac85ba6b:303b72420000001f (host=vb0234.foo:22000): Averaged Fragment F06: split sizes: min: 25.70 MB, max: 26.47 MB, avg: 25.96 MB, stddev: 363.28 KB completion times: min:381.822ms max:384.457ms mean: 383.817ms stddev:744.653us execution rates: min:66.85 MB/sec max:69.32 MB/sec mean:67.63 MB/sec stddev:1.01 MB/sec num instances: 9 Fragment F06: Instance d7448d78ac85ba6b:303b72420000003f (host=vb0206.foo:22000): Instance d7448d78ac85ba6b:303b724200000043 (host=vb0208.foo:22000): Instance d7448d78ac85ba6b:303b724200000044 (host=vb0214.foo:22000): Instance d7448d78ac85ba6b:303b72420000003d (host=vb0216.foo:22000): Instance d7448d78ac85ba6b:303b724200000040 (host=vb0218.foo:22000): Instance d7448d78ac85ba6b:303b72420000003e (host=vb0222.foo:22000): Instance d7448d78ac85ba6b:303b724200000045 (host=vb0224.foo:22000): Instance d7448d78ac85ba6b:303b724200000042 (host=vb0230.foo:22000): Instance d7448d78ac85ba6b:303b724200000041 (host=vb0232.foo:22000): Averaged Fragment F07: split sizes: min: 4.21 GB, max: 4.71 GB, avg: 4.60 GB, stddev: 147.33 MB completion times: min:381.821ms max:384.900ms mean: 383.914ms stddev:798.573us execution rates: min:10.93 GB/sec max:12.31 GB/sec mean:11.97 GB/sec stddev:395.03 MB/sec num instances: 14 Fragment F07: Instance d7448d78ac85ba6b:303b72420000003c (host=vb0204.foo:22000): Instance d7448d78ac85ba6b:303b724200000034 (host=vb0206.foo:22000): Instance d7448d78ac85ba6b:303b724200000035 (host=vb0208.foo:22000): Instance d7448d78ac85ba6b:303b724200000031 (host=vb0212.foo:22000): Instance d7448d78ac85ba6b:303b724200000032 (host=vb0214.foo:22000): Instance d7448d78ac85ba6b:303b724200000030 (host=vb0216.foo:22000): Instance d7448d78ac85ba6b:303b72420000003b (host=vb0218.foo:22000): Instance d7448d78ac85ba6b:303b724200000039 (host=vb0220.foo:22000): Instance d7448d78ac85ba6b:303b724200000038 (host=vb0222.foo:22000): Instance d7448d78ac85ba6b:303b724200000033 (host=vb0224.foo:22000): Instance d7448d78ac85ba6b:303b72420000003a (host=vb0228.foo:22000): Instance d7448d78ac85ba6b:303b72420000002f (host=vb0230.foo:22000): Instance d7448d78ac85ba6b:303b724200000037 (host=vb0232.foo:22000): Instance d7448d78ac85ba6b:303b724200000036 (host=vb0234.foo:22000):