Query (id=564aad669bd61da1:2921b0400000000): Summary: Session ID: d4aae3065228126:c1fb4f0d4b4a8e84 Session Type: BEESWAX Start Time: 2018-01-03 14:46:47.996040000 End Time: 2018-01-03 14:47:12.572503000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.10.0-cdh5.13.0 RELEASE (build 21d938d2a680a4b0708992fc0f23d3d156c2a355) User: mmokhtar Connected User: mmokhtar Delegated User: Network Address: ::ffff:10.20.122.102:60862 Default Db: test_db Sql Statement: select count(*),max(hour + nb_display) from bi_arbitrage_full Coordinator: d2402.foo.com:22000 Query Options (set by configuration): Query Options (set by configuration and planner): MT_DOP=0 Plan: ---------------- Max Per-Host Resource Reservation: Memory=0B Per-Host Resource Estimates: Memory=52.00MB WARNING: The following tables are missing relevant table and/or column statistics. test_db.bi_arbitrage_full F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 03:AGGREGATE [FINALIZE] | output: count:merge(*), max:merge(hour + nb_display) | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB | tuple-ids=1 row-size=16B cardinality=1 | 02:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=1 row-size=16B cardinality=1 | F00:PLAN FRAGMENT [RANDOM] hosts=10 instances=10 Per-Host Resources: mem-estimate=42.00MB mem-reservation=0B 01:AGGREGATE | output: count(*), max(hour + nb_display) | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB | tuple-ids=1 row-size=16B cardinality=1 | 00:SCAN HDFS [test_db.bi_arbitrage_full, RANDOM] partitions=186/186 files=136550 size=14.97GB stats-rows=unavailable extrapolated-rows=disabled table stats: rows=unavailable size=unavailable column stats: unavailable mem-estimate=32.00MB mem-reservation=0B tuple-ids=0 row-size=8B cardinality=unavailable ---------------- Estimated Per-Host Mem: 54525952 Tables Missing Stats: test_db.bi_arbitrage_full Per Host Min Reservation: d2402.foo.com:22000(0) d2403.foo.com:22000(0) d2404.foo.com:22000(0) d2405.foo.com:22000(0) d2406.foo.com:22000(0) d2407.foo.com:22000(0) d2408.foo.com:22000(0) d2409.foo.com:22000(0) d2410.foo.com:22000(0) d2411.foo.com:22000(0) Request Pool: root.mmokhtar Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail ------------------------------------------------------------------------------------------------------------------- 03:AGGREGATE 1 446.439us 446.439us 1 1 56.00 KB 10.00 MB FINALIZE 02:EXCHANGE 1 23s392ms 23s392ms 10 1 0 0 UNPARTITIONED 01:AGGREGATE 10 1s269ms 1s326ms 10 1 185.25 KB 10.00 MB 00:SCAN HDFS 10 20s678ms 22s098ms 1.89B -1 23.78 MB 32.00 MB test_db.bi_arbitrage_full Errors: Planner Timeline: 159.195ms - Analysis finished: 3.920ms (3.920ms) - Equivalence classes computed: 4.161ms (240.242us) - Single node plan created: 154.524ms (150.363ms) - Runtime filters computed: 154.572ms (47.734us) - Distributed plan created: 155.357ms (784.974us) - Lineage info computed: 155.580ms (223.458us) - Planning finished: 159.195ms (3.614ms) Query Timeline: 24s580ms - Query submitted: 108.564us (108.564us) - Planning finished: 522.789ms (522.680ms) - Submit for admission: 865.744ms (342.955ms) - Completed admission: 866.059ms (314.543us) - Ready to start on 10 backends: 911.521ms (45.462ms) - All 10 execution backends (11 fragment instances) started: 940.280ms (28.758ms) - Rows available: 24s350ms (23s410ms) - First row fetched: 24s572ms (222.041ms) - Unregister query: 24s576ms (3.574ms) - ComputeScanRangeAssignmentTimer: 222.850ms ImpalaServer: - ClientFetchWaitTimer: 223.599ms - RowMaterializationTimer: 1.982ms Execution Profile 564aad669bd61da1:2921b0400000000:(Total: 23s431ms, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 0 Filter routing table: ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled ------------------------------------------------------------------------------------------------------------------- Backend startup latencies: Count: 10, min / max: 18ms / 21ms, 25th %-ile: 18ms, 50th %-ile: 19ms, 75th %-ile: 20ms, 90th %-ile: 20ms, 95th %-ile: 21ms, 99.9th %-ile: 21ms Per Node Peak Memory Usage: d2402.foo.com:22000(19.08 MB) d2403.foo.com:22000(18.54 MB) d2404.foo.com:22000(23.82 MB) d2405.foo.com:22000(20.91 MB) d2406.foo.com:22000(19.26 MB) d2407.foo.com:22000(19.18 MB) d2408.foo.com:22000(20.36 MB) d2409.foo.com:22000(17.53 MB) d2410.foo.com:22000(16.16 MB) d2411.foo.com:22000(18.10 MB) - FiltersReceived: 0 (0) - FinalizationTimer: 0.000ns Averaged Fragment F01:(Total: 23s637ms, non-child: 222.765ms, % non-child: 0.94%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:23s638ms max:23s638ms mean: 23s638ms stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 64.02 KB (65552) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 19.08 MB (20007728) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 23s392ms - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 1 (1) - TotalThreadsTotalWallClockTime: 23s614ms - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 871.000us - TotalThreadsVoluntaryContextSwitches: 13 (13) Fragment Instance Lifecycle Timings: - ExecTime: 222.216ms - ExecTreeExecTime: 7.554us - OpenTime: 23s392ms - ExecTreeOpenTime: 23s392ms - PrepareTime: 22.318ms - ExecTreePrepareTime: 71.726us PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 21.762ms, non-child: 21.762ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 21.203ms AGGREGATION_NODE (id=3):(Total: 23s392ms, non-child: 446.439us, % non-child: 0.00%) - BuildTime: 212.320us - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 56.00 KB (57344) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) EXCHANGE_NODE (id=2):(Total: 23s392ms, non-child: 23s392ms, % non-child: 100.00%) - ConvertRowBatchTime: 37.539us - PeakMemoryUsage: 0 - RowsReturned: 10 (10) - RowsReturnedRate: 0 DataStreamReceiver: - BytesReceived: 250.00 B (250) - DeserializeRowBatchTimer: 72.967us - FirstBatchArrivalWaitTime: 19s797ms - PeakMemoryUsage: 4.02 KB (4112) - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Coordinator Fragment F01: Instance 564aad669bd61da1:2921b0400000000 (host=d2402.foo.com:22000):(Total: 23s637ms, non-child: 222.765ms, % non-child: 0.94%) MemoryUsage(500.000ms): 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 12.00 KB, 28.01 KB, 28.01 KB, 36.01 KB, 40.01 KB, 44.01 KB, 56.01 KB, 60.01 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 64.02 KB (65552) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 19.08 MB (20007728) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 23s392ms - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 1 (1) - TotalThreadsTotalWallClockTime: 23s614ms - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 871.000us - TotalThreadsVoluntaryContextSwitches: 13 (13) Fragment Instance Lifecycle Timings: - ExecTime: 222.216ms - ExecTreeExecTime: 7.554us - OpenTime: 23s392ms - ExecTreeOpenTime: 23s392ms - PrepareTime: 22.318ms - ExecTreePrepareTime: 71.726us PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 21.762ms, non-child: 21.762ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 21.203ms AGGREGATION_NODE (id=3):(Total: 23s392ms, non-child: 446.439us, % non-child: 0.00%) - BuildTime: 212.320us - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 56.00 KB (57344) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) EXCHANGE_NODE (id=2):(Total: 23s392ms, non-child: 23s392ms, % non-child: 100.00%) - ConvertRowBatchTime: 37.539us - PeakMemoryUsage: 0 - RowsReturned: 10 (10) - RowsReturnedRate: 0 DataStreamReceiver: BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 25.00 B, 25.00 B, 75.00 B, 100.00 B, 125.00 B, 200.00 B, 225.00 B - BytesReceived: 250.00 B (250) - DeserializeRowBatchTimer: 72.967us - FirstBatchArrivalWaitTime: 19s797ms - PeakMemoryUsage: 4.02 KB (4112) - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F00:(Total: 22s005ms, non-child: 2.772ms, % non-child: 0.01%) split sizes: min: 1.50 GB, max: 1.50 GB, avg: 1.50 GB, stddev: 1.13 MB completion times: min:19s818ms max:23s416ms mean: 22s006ms stddev:1s014ms execution rates: min:65.43 MB/sec max:77.31 MB/sec mean:69.82 MB/sec stddev:3.33 MB/sec num instances: 10 - AverageThreadTokens: 24.84 - BloomFilterBytes: 0 - PeakMemoryUsage: 19.29 MB (20230922) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 19.30 MB (20232561) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 248.541us - TotalStorageWaitTime: 7m35s - TotalThreadsInvoluntaryContextSwitches: 1.80K (1799) - TotalThreadsTotalWallClockTime: 9m6s - TotalThreadsSysTime: 1s441ms - TotalThreadsUserTime: 8s358ms - TotalThreadsVoluntaryContextSwitches: 248.60K (248596) Fragment Instance Lifecycle Timings: - ExecTime: 402.777us - ExecTreeExecTime: 9.265us - OpenTime: 21s944ms - ExecTreeOpenTime: 21s910ms - PrepareTime: 60.518ms - ExecTreePrepareTime: 36.522ms DataStreamSender (dst_id=2):(Total: 305.704us, non-child: 305.704us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 950.81 KB/sec - OverallThroughput: 84.03 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 12.214us - TransmitDataRPCTime: 26.584us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 54.580ms, non-child: 54.580ms, % non-child: 100.00%) - CodegenTime: 2.413ms - CompileTime: 11.213ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 21.399ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 21.353ms AGGREGATION_NODE (id=1):(Total: 21s947ms, non-child: 1s269ms, % non-child: 5.78%) - BuildTime: 1s151ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 178.95 KB (183244) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 20s678ms, non-child: 20s678ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.16 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.32 GB (1413061946) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 40.97K (40965) - DecompressionTime: 324.901ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 13.65K (13655) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 19.26 MB (20191941) - PerReadThreadRawHdfsThroughput: 344.58 MB/sec - RemoteScanRanges: 122.89K (122895) - RowBatchQueueGetWaitTime: 20s089ms - RowBatchQueuePutWaitTime: 1.122ms - RowsRead: 189.00M (188999351) - RowsReturned: 189.00M (188999351) - RowsReturnedRate: 9.17 M/sec - ScanRangesComplete: 13.65K (13655) - ScannerThreadsInvoluntaryContextSwitches: 1.39K (1391) - ScannerThreadsTotalWallClockTime: 8m44s - MaterializeTupleTime(*): 3s074ms - ScannerThreadsSysTime: 1s132ms - ScannerThreadsUserTime: 6s304ms - ScannerThreadsVoluntaryContextSwitches: 100.47K (100475) - TotalRawHdfsReadTime(*): 3s912ms - TotalReadThroughput: 60.49 MB/sec Fragment F00: Instance 564aad669bd61da1:2921b0400000002 (host=d2403.foo.com:22000):(Total: 23s415ms, non-child: 2.531ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:14594/1.50 GB MemoryUsage(500.000ms): 792.23 KB, 736.24 KB, 1.02 MB, 976.28 KB, 1.27 MB, 10.32 MB, 980.29 KB, 832.27 KB, 768.27 KB, 1.07 MB, 616.25 KB, 5.36 MB, 4.73 MB, 972.28 KB, 864.24 KB, 5.44 MB, 4.79 MB, 616.25 KB, 688.25 KB, 5.05 MB, 9.59 MB, 1.01 MB, 9.30 MB, 900.27 KB, 1.09 MB, 836.27 KB, 648.28 KB, 640.27 KB, 892.26 KB, 1.07 MB, 1.07 MB, 608.24 KB, 648.28 KB, 1.13 MB, 748.25 KB, 980.29 KB, 5.53 MB, 1.01 MB, 1.21 MB, 632.26 KB, 664.30 KB, 5.27 MB, 4.93 MB, 632.26 KB, 900.27 KB, 808.25 KB, 896.27 KB ThreadUsage(500.000ms): 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 25.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 18.54 MB (19439289) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 18.54 MB (19443394) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 179.349us - TotalStorageWaitTime: 8m1s - TotalThreadsInvoluntaryContextSwitches: 1.91K (1907) - TotalThreadsTotalWallClockTime: 9m42s - TotalThreadsSysTime: 1s501ms - TotalThreadsUserTime: 8s409ms - TotalThreadsVoluntaryContextSwitches: 244.64K (244636) Fragment Instance Lifecycle Timings: - ExecTime: 289.317us - ExecTreeExecTime: 5.717us - OpenTime: 23s356ms - ExecTreeOpenTime: 23s325ms - PrepareTime: 58.413ms - ExecTreePrepareTime: 35.590ms DataStreamSender (dst_id=2):(Total: 279.883us, non-child: 279.883us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 771.47 KB/sec - OverallThroughput: 87.23 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 10.307us - TransmitDataRPCTime: 31.646us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 51.909ms, non-child: 51.909ms, % non-child: 100.00%) - CodegenTime: 2.241ms - CompileTime: 10.868ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 20.141ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 20.288ms AGGREGATION_NODE (id=1):(Total: 23s360ms, non-child: 1s262ms, % non-child: 5.40%) ExecOption: Codegen Enabled - BuildTime: 1s144ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 176.25 KB (180480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 22s098ms, non-child: 22s098ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:14594/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 14594 out of 14594 Hdfs Read Thread Concurrency Bucket: 0:91.49% 1:8.511% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:29188 BytesRead(500.000ms): 980.63 KB, 36.68 MB, 73.00 MB, 95.09 MB, 110.33 MB, 137.01 MB, 165.93 MB, 192.61 MB, 226.33 MB, 264.54 MB, 288.27 MB, 326.68 MB, 350.42 MB, 392.39 MB, 424.13 MB, 445.96 MB, 468.92 MB, 504.94 MB, 526.36 MB, 549.62 MB, 582.63 MB, 613.21 MB, 636.76 MB, 658.40 MB, 681.93 MB, 710.23 MB, 733.89 MB, 783.48 MB, 809.78 MB, 845.46 MB, 876.23 MB, 898.04 MB, 910.62 MB, 950.88 MB, 972.79 MB, 992.02 MB, 1018.23 MB, 1.04 GB, 1.08 GB, 1.10 GB, 1.12 GB, 1.15 GB, 1.19 GB, 1.22 GB, 1.25 GB, 1.28 GB, 1.31 GB - FooterProcessingTime: (Avg: 15.454ms ; Min: 912.412us ; Max: 334.021ms ; Number of samples: 14594) - AverageHdfsReadThreadConcurrency: 0.09 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.33 GB (1424815870) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 43.78K (43782) - DecompressionTime: 311.014ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 14.59K (14594) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 18.51 MB (19406350) - PerReadThreadRawHdfsThroughput: 348.31 MB/sec - RemoteScanRanges: 131.35K (131346) - RowBatchQueueGetWaitTime: 21s518ms - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 182.94M (182937778) - RowsReturned: 182.94M (182937778) - RowsReturnedRate: 8.28 M/sec - ScanRangesComplete: 14.59K (14594) - ScannerThreadsInvoluntaryContextSwitches: 1.53K (1533) - ScannerThreadsTotalWallClockTime: 9m18s - MaterializeTupleTime(*): 2s964ms - ScannerThreadsSysTime: 1s201ms - ScannerThreadsUserTime: 6s383ms - ScannerThreadsVoluntaryContextSwitches: 106.42K (106424) - TotalRawHdfsReadTime(*): 3s901ms - TotalReadThroughput: 57.26 MB/sec Instance 564aad669bd61da1:2921b0400000003 (host=d2410.foo.com:22000):(Total: 23s062ms, non-child: 2.673ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:14363/1.50 GB MemoryUsage(500.000ms): 544.17 KB, 5.23 MB, 5.02 MB, 648.28 KB, 1.11 MB, 800.24 KB, 1.00 MB, 936.25 KB, 760.26 KB, 760.26 KB, 916.29 KB, 952.26 KB, 624.25 KB, 656.29 KB, 640.27 KB, 932.24 KB, 5.54 MB, 1.16 MB, 1.21 MB, 868.31 KB, 632.26 KB, 648.28 KB, 776.28 KB, 712.28 KB, 4.82 MB, 5.54 MB, 880.25 KB, 5.12 MB, 632.26 KB, 828.26 KB, 9.87 MB, 5.12 MB, 4.95 MB, 648.28 KB, 704.27 KB, 4.86 MB, 1.13 MB, 648.28 KB, 824.26 KB, 5.08 MB, 780.28 KB, 896.26 KB, 640.27 KB, 956.26 KB, 5.72 MB, 844.28 KB ThreadUsage(500.000ms): 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 25.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.16 MB (16942577) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 16.16 MB (16942577) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 228.234us - TotalStorageWaitTime: 7m57s - TotalThreadsInvoluntaryContextSwitches: 1.89K (1894) - TotalThreadsTotalWallClockTime: 9m32s - TotalThreadsSysTime: 1s446ms - TotalThreadsUserTime: 8s412ms - TotalThreadsVoluntaryContextSwitches: 256.88K (256875) Fragment Instance Lifecycle Timings: - ExecTime: 363.041us - ExecTreeExecTime: 11.746us - OpenTime: 22s998ms - ExecTreeOpenTime: 22s965ms - PrepareTime: 62.684ms - ExecTreePrepareTime: 37.509ms DataStreamSender (dst_id=2):(Total: 245.635us, non-child: 245.635us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 944.38 KB/sec - OverallThroughput: 99.39 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 31.669us - TransmitDataRPCTime: 25.852us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 56.609ms, non-child: 56.609ms, % non-child: 100.00%) - CodegenTime: 2.348ms - CompileTime: 11.469ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 21.901ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 22.615ms AGGREGATION_NODE (id=1):(Total: 23s002ms, non-child: 1s233ms, % non-child: 5.36%) ExecOption: Codegen Enabled - BuildTime: 1s117ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 185.25 KB (189696) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 21s768ms, non-child: 21s768ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:14363/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 14363 out of 14363 Hdfs Read Thread Concurrency Bucket: 0:84.78% 1:15.22% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:28726 BytesRead(500.000ms): 0, 25.81 MB, 53.67 MB, 74.35 MB, 95.24 MB, 115.46 MB, 141.25 MB, 162.96 MB, 194.96 MB, 220.64 MB, 252.17 MB, 279.54 MB, 305.79 MB, 337.75 MB, 363.80 MB, 383.20 MB, 411.19 MB, 446.18 MB, 477.76 MB, 500.85 MB, 527.76 MB, 559.97 MB, 592.66 MB, 617.53 MB, 661.55 MB, 696.33 MB, 731.30 MB, 760.49 MB, 790.47 MB, 823.12 MB, 862.10 MB, 898.85 MB, 922.10 MB, 948.57 MB, 990.11 MB, 1018.92 MB, 1.03 GB, 1.06 GB, 1.08 GB, 1.11 GB, 1.14 GB, 1.18 GB, 1.21 GB, 1.23 GB, 1.26 GB, 1.29 GB - FooterProcessingTime: (Avg: 15.781ms ; Min: 1.616ms ; Max: 339.578ms ; Number of samples: 14363) - AverageHdfsReadThreadConcurrency: 0.15 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.32 GB (1421616203) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 43.09K (43089) - DecompressionTime: 317.219ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 14.36K (14363) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 16.12 MB (16900422) - PerReadThreadRawHdfsThroughput: 360.07 MB/sec - RemoteScanRanges: 129.27K (129267) - RowBatchQueueGetWaitTime: 21s182ms - RowBatchQueuePutWaitTime: 1.651ms - RowsRead: 184.56M (184556009) - RowsReturned: 184.56M (184556009) - RowsReturnedRate: 8.48 M/sec - ScanRangesComplete: 14.36K (14363) - ScannerThreadsInvoluntaryContextSwitches: 1.46K (1461) - ScannerThreadsTotalWallClockTime: 9m10s - MaterializeTupleTime(*): 2s953ms - ScannerThreadsSysTime: 1s139ms - ScannerThreadsUserTime: 6s365ms - ScannerThreadsVoluntaryContextSwitches: 105.58K (105577) - TotalRawHdfsReadTime(*): 3s765ms - TotalReadThroughput: 57.49 MB/sec Instance 564aad669bd61da1:2921b0400000007 (host=d2411.foo.com:22000):(Total: 22s682ms, non-child: 3.203ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:14126/1.50 GB MemoryUsage(500.000ms): 1.08 MB, 5.22 MB, 980.29 KB, 5.11 MB, 956.26 KB, 632.26 KB, 648.28 KB, 1.07 MB, 1012.25 KB, 720.29 KB, 884.25 KB, 1.22 MB, 928.23 KB, 5.46 MB, 860.23 KB, 1008.25 KB, 5.22 MB, 772.27 KB, 5.04 MB, 624.25 KB, 768.27 KB, 5.15 MB, 5.15 MB, 9.88 MB, 880.25 KB, 680.25 KB, 1.16 MB, 1.13 MB, 5.12 MB, 1001.28 KB, 696.26 KB, 728.23 KB, 1.02 MB, 4.81 MB, 984.29 KB, 1.13 MB, 992.23 KB, 5.58 MB, 10.06 MB, 4.83 MB, 648.28 KB, 4.99 MB, 864.24 KB, 1000.25 KB, 904.28 KB ThreadUsage(500.000ms): 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 25.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 18.10 MB (18983400) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 18.10 MB (18983400) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 307.683us - TotalStorageWaitTime: 7m47s - TotalThreadsInvoluntaryContextSwitches: 1.89K (1893) - TotalThreadsTotalWallClockTime: 9m23s - TotalThreadsSysTime: 1s430ms - TotalThreadsUserTime: 8s266ms - TotalThreadsVoluntaryContextSwitches: 254.31K (254306) Fragment Instance Lifecycle Timings: - ExecTime: 444.470us - ExecTreeExecTime: 5.003us - OpenTime: 22s616ms - ExecTreeOpenTime: 22s582ms - PrepareTime: 65.496ms - ExecTreePrepareTime: 40.810ms DataStreamSender (dst_id=2):(Total: 235.485us, non-child: 235.485us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 1.04 MB/sec - OverallThroughput: 103.67 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 11.875us - TransmitDataRPCTime: 22.999us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 55.837ms, non-child: 55.837ms, % non-child: 100.00%) - CodegenTime: 2.916ms - CompileTime: 11.552ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 21.964ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 21.639ms AGGREGATION_NODE (id=1):(Total: 22s623ms, non-child: 1s202ms, % non-child: 5.31%) ExecOption: Codegen Enabled - BuildTime: 1s087ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 185.25 KB (189696) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 21s420ms, non-child: 21s420ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:14126/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 14126 out of 14126 Hdfs Read Thread Concurrency Bucket: 0:80% 1:20% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:28252 BytesRead(500.000ms): 16.89 MB, 48.97 MB, 76.00 MB, 118.61 MB, 124.24 MB, 153.10 MB, 182.70 MB, 201.79 MB, 246.01 MB, 277.79 MB, 306.31 MB, 349.32 MB, 384.80 MB, 401.81 MB, 432.96 MB, 457.51 MB, 495.60 MB, 530.17 MB, 553.11 MB, 594.30 MB, 614.89 MB, 644.65 MB, 688.02 MB, 723.06 MB, 739.31 MB, 777.82 MB, 811.35 MB, 839.55 MB, 864.84 MB, 902.85 MB, 923.14 MB, 953.92 MB, 971.41 MB, 998.26 MB, 1.01 GB, 1.04 GB, 1.07 GB, 1.11 GB, 1.14 GB, 1.15 GB, 1.18 GB, 1.20 GB, 1.22 GB, 1.25 GB, 1.30 GB - FooterProcessingTime: (Avg: 15.729ms ; Min: 3.037ms ; Max: 335.806ms ; Number of samples: 14126) - AverageHdfsReadThreadConcurrency: 0.20 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.32 GB (1418956564) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 42.38K (42378) - DecompressionTime: 320.303ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 14.13K (14126) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 18.07 MB (18950461) - PerReadThreadRawHdfsThroughput: 338.54 MB/sec - RemoteScanRanges: 127.13K (127134) - RowBatchQueueGetWaitTime: 20s841ms - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 185.72M (185719848) - RowsReturned: 185.72M (185719848) - RowsReturnedRate: 8.67 M/sec - ScanRangesComplete: 14.13K (14126) - ScannerThreadsInvoluntaryContextSwitches: 1.46K (1456) - ScannerThreadsTotalWallClockTime: 9m1s - MaterializeTupleTime(*): 2s998ms - ScannerThreadsSysTime: 1s133ms - ScannerThreadsUserTime: 6s262ms - ScannerThreadsVoluntaryContextSwitches: 104.54K (104544) - TotalRawHdfsReadTime(*): 3s997ms - TotalReadThroughput: 59.02 MB/sec Instance 564aad669bd61da1:2921b0400000004 (host=d2405.foo.com:22000):(Total: 22s524ms, non-child: 2.557ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:13914/1.50 GB MemoryUsage(500.000ms): 11.52 KB, 664.30 KB, 672.31 KB, 648.28 KB, 1.04 MB, 1.02 MB, 4.89 MB, 700.26 KB, 964.27 KB, 5.14 MB, 5.16 MB, 852.29 KB, 9.69 MB, 624.25 KB, 1.01 MB, 720.29 KB, 768.27 KB, 1.08 MB, 1008.25 KB, 1.18 MB, 916.29 KB, 4.81 MB, 960.26 KB, 5.37 MB, 656.29 KB, 5.09 MB, 9.32 MB, 904.28 KB, 5.02 MB, 656.29 KB, 968.27 KB, 892.26 KB, 816.25 KB, 648.28 KB, 5.42 MB, 584.21 KB, 720.29 KB, 872.25 KB, 5.29 MB, 1.16 MB, 996.31 KB, 5.08 MB, 1.00 MB, 4.93 MB, 768.27 KB ThreadUsage(500.000ms): 1, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 24.47 - BloomFilterBytes: 0 - PeakMemoryUsage: 20.91 MB (21928928) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 20.91 MB (21928928) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 249.989us - TotalStorageWaitTime: 7m46s - TotalThreadsInvoluntaryContextSwitches: 1.90K (1900) - TotalThreadsTotalWallClockTime: 9m19s - TotalThreadsSysTime: 1s499ms - TotalThreadsUserTime: 8s386ms - TotalThreadsVoluntaryContextSwitches: 247.81K (247812) Fragment Instance Lifecycle Timings: - ExecTime: 389.734us - ExecTreeExecTime: 6.796us - OpenTime: 22s463ms - ExecTreeOpenTime: 22s430ms - PrepareTime: 60.873ms - ExecTreePrepareTime: 37.218ms DataStreamSender (dst_id=2):(Total: 227.320us, non-child: 227.320us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 956.93 KB/sec - OverallThroughput: 107.40 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 12.150us - TransmitDataRPCTime: 25.513us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 54.151ms, non-child: 54.151ms, % non-child: 100.00%) - CodegenTime: 2.296ms - CompileTime: 10.810ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 21.496ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 21.264ms AGGREGATION_NODE (id=1):(Total: 22s467ms, non-child: 1s326ms, % non-child: 5.91%) ExecOption: Codegen Enabled - BuildTime: 1s200ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 176.25 KB (180480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 21s140ms, non-child: 21s140ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:13914/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 13914 out of 13914 Hdfs Read Thread Concurrency Bucket: 0:81.82% 1:18.18% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:27828 BytesRead(500.000ms): 0, 38.04 MB, 61.71 MB, 86.24 MB, 111.63 MB, 136.20 MB, 173.13 MB, 198.69 MB, 230.42 MB, 257.21 MB, 288.14 MB, 318.14 MB, 339.87 MB, 383.71 MB, 411.91 MB, 441.61 MB, 472.75 MB, 493.39 MB, 519.45 MB, 559.56 MB, 575.69 MB, 611.24 MB, 648.27 MB, 684.65 MB, 716.86 MB, 753.52 MB, 791.08 MB, 812.62 MB, 839.56 MB, 880.57 MB, 918.73 MB, 936.68 MB, 963.97 MB, 995.09 MB, 1.00 GB, 1.03 GB, 1.06 GB, 1.09 GB, 1.12 GB, 1.15 GB, 1.17 GB, 1.20 GB, 1.24 GB, 1.27 GB, 1.29 GB - FooterProcessingTime: (Avg: 15.868ms ; Min: 2.166ms ; Max: 341.651ms ; Number of samples: 13914) - AverageHdfsReadThreadConcurrency: 0.20 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.32 GB (1417546388) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 41.74K (41742) - DecompressionTime: 320.810ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 13.91K (13914) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 20.87 MB (21886773) - PerReadThreadRawHdfsThroughput: 338.93 MB/sec - RemoteScanRanges: 125.23K (125226) - RowBatchQueueGetWaitTime: 20s540ms - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 187.72M (187716424) - RowsReturned: 187.72M (187716424) - RowsReturnedRate: 8.88 M/sec - ScanRangesComplete: 13.91K (13914) - ScannerThreadsInvoluntaryContextSwitches: 1.52K (1525) - ScannerThreadsTotalWallClockTime: 8m56s - MaterializeTupleTime(*): 3s061ms - ScannerThreadsSysTime: 1s181ms - ScannerThreadsUserTime: 6s303ms - ScannerThreadsVoluntaryContextSwitches: 102.41K (102406) - TotalRawHdfsReadTime(*): 3s988ms - TotalReadThroughput: 58.72 MB/sec Instance 564aad669bd61da1:2921b0400000008 (host=d2407.foo.com:22000):(Total: 22s496ms, non-child: 2.629ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:13852/1.50 GB MemoryUsage(500.000ms): 19.52 KB, 640.27 KB, 760.26 KB, 844.28 KB, 1.05 MB, 884.25 KB, 1.03 MB, 624.25 KB, 904.27 KB, 872.25 KB, 1.16 MB, 5.41 MB, 1.00 MB, 1.02 MB, 5.05 MB, 720.29 KB, 608.24 KB, 964.27 KB, 688.25 KB, 948.25 KB, 5.19 MB, 9.75 MB, 5.18 MB, 5.18 MB, 624.25 KB, 640.27 KB, 768.27 KB, 14.18 MB, 4.80 MB, 608.24 KB, 912.28 KB, 1.21 MB, 728.30 KB, 640.27 KB, 5.11 MB, 5.52 MB, 5.15 MB, 4.84 MB, 5.36 MB, 1008.25 KB, 708.27 KB, 5.18 MB, 5.18 MB, 656.22 KB, 824.26 KB ThreadUsage(500.000ms): 1, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 24.47 - BloomFilterBytes: 0 - PeakMemoryUsage: 19.18 MB (20110588) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 19.18 MB (20110588) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 276.712us - TotalStorageWaitTime: 7m44s - TotalThreadsInvoluntaryContextSwitches: 1.92K (1918) - TotalThreadsTotalWallClockTime: 9m18s - TotalThreadsSysTime: 1s460ms - TotalThreadsUserTime: 8s460ms - TotalThreadsVoluntaryContextSwitches: 251.69K (251686) Fragment Instance Lifecycle Timings: - ExecTime: 430.981us - ExecTreeExecTime: 6.864us - OpenTime: 22s435ms - ExecTreeOpenTime: 22s403ms - PrepareTime: 59.905ms - ExecTreePrepareTime: 36.488ms DataStreamSender (dst_id=2):(Total: 504.629us, non-child: 504.629us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 1.07 MB/sec - OverallThroughput: 48.38 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 10.955us - TransmitDataRPCTime: 22.310us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 53.230ms, non-child: 53.230ms, % non-child: 100.00%) - CodegenTime: 2.188ms - CompileTime: 10.848ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 20.821ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 20.959ms AGGREGATION_NODE (id=1):(Total: 22s439ms, non-child: 1s266ms, % non-child: 5.64%) ExecOption: Codegen Enabled - BuildTime: 1s148ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 176.25 KB (180480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 21s173ms, non-child: 21s173ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:13852/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 13852 out of 13852 Hdfs Read Thread Concurrency Bucket: 0:88.64% 1:11.36% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:27704 BytesRead(500.000ms): 0, 28.50 MB, 49.77 MB, 85.51 MB, 98.10 MB, 117.72 MB, 141.24 MB, 171.67 MB, 206.83 MB, 242.11 MB, 275.24 MB, 302.62 MB, 330.79 MB, 352.80 MB, 385.80 MB, 403.35 MB, 433.10 MB, 465.41 MB, 494.31 MB, 517.15 MB, 545.24 MB, 592.05 MB, 630.72 MB, 667.69 MB, 696.51 MB, 721.78 MB, 748.07 MB, 785.57 MB, 813.40 MB, 845.92 MB, 877.98 MB, 912.48 MB, 953.20 MB, 976.85 MB, 1009.78 MB, 1.02 GB, 1.06 GB, 1.10 GB, 1.12 GB, 1.15 GB, 1.18 GB, 1.22 GB, 1.25 GB, 1.28 GB, 1.30 GB - FooterProcessingTime: (Avg: 15.831ms ; Min: 2.375ms ; Max: 341.572ms ; Number of samples: 13852) - AverageHdfsReadThreadConcurrency: 0.11 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.32 GB (1416350504) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 41.56K (41556) - DecompressionTime: 326.425ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 13.85K (13852) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 19.15 MB (20076625) - PerReadThreadRawHdfsThroughput: 354.14 MB/sec - RemoteScanRanges: 124.67K (124668) - RowBatchQueueGetWaitTime: 20s574ms - RowBatchQueuePutWaitTime: 486.126us - RowsRead: 188.26M (188262493) - RowsReturned: 188.26M (188262493) - RowsReturnedRate: 8.89 M/sec - ScanRangesComplete: 13.85K (13852) - ScannerThreadsInvoluntaryContextSwitches: 1.49K (1489) - ScannerThreadsTotalWallClockTime: 8m56s - MaterializeTupleTime(*): 3s070ms - ScannerThreadsSysTime: 1s162ms - ScannerThreadsUserTime: 6s380ms - ScannerThreadsVoluntaryContextSwitches: 102.12K (102118) - TotalRawHdfsReadTime(*): 3s814ms - TotalReadThroughput: 59.16 MB/sec Instance 564aad669bd61da1:2921b040000000a (host=d2409.foo.com:22000):(Total: 22s009ms, non-child: 3.055ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:13776/1.50 GB MemoryUsage(500.000ms): 0, 1.13 MB, 976.28 KB, 5.25 MB, 912.28 KB, 656.29 KB, 4.73 MB, 1.07 MB, 784.29 KB, 632.26 KB, 696.26 KB, 632.26 KB, 1.15 MB, 5.70 MB, 1.08 MB, 976.28 KB, 4.89 MB, 5.15 MB, 1.21 MB, 816.25 KB, 10.17 MB, 4.90 MB, 5.30 MB, 1.05 MB, 5.49 MB, 984.29 KB, 608.24 KB, 724.29 KB, 900.27 KB, 948.25 KB, 892.26 KB, 824.26 KB, 648.28 KB, 1.01 MB, 1.33 MB, 5.18 MB, 1.29 MB, 5.49 MB, 904.27 KB, 936.25 KB, 656.29 KB, 908.28 KB, 808.25 KB, 1.02 MB ThreadUsage(500.000ms): 1, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 24.45 - BloomFilterBytes: 0 - PeakMemoryUsage: 17.53 MB (18386694) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 17.53 MB (18386694) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 369.314us - TotalStorageWaitTime: 7m37s - TotalThreadsInvoluntaryContextSwitches: 1.66K (1657) - TotalThreadsTotalWallClockTime: 9m6s - TotalThreadsSysTime: 1s512ms - TotalThreadsUserTime: 8s184ms - TotalThreadsVoluntaryContextSwitches: 252.09K (252090) Fragment Instance Lifecycle Timings: - ExecTime: 615.411us - ExecTreeExecTime: 12.911us - OpenTime: 21s950ms - ExecTreeOpenTime: 21s916ms - PrepareTime: 59.027ms - ExecTreePrepareTime: 35.416ms DataStreamSender (dst_id=2):(Total: 287.510us, non-child: 287.510us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 964.22 KB/sec - OverallThroughput: 84.92 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 9.133us - TransmitDataRPCTime: 25.320us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 53.982ms, non-child: 53.982ms, % non-child: 100.00%) - CodegenTime: 2.658ms - CompileTime: 11.546ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 20.995ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 20.870ms AGGREGATION_NODE (id=1):(Total: 21s952ms, non-child: 1s253ms, % non-child: 5.71%) ExecOption: Codegen Enabled - BuildTime: 1s137ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 185.25 KB (189696) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 20s699ms, non-child: 20s699ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:13776/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 13776 out of 13776 Hdfs Read Thread Concurrency Bucket: 0:90.7% 1:9.302% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:27552 BytesRead(500.000ms): 0, 23.57 MB, 48.23 MB, 77.45 MB, 107.36 MB, 124.30 MB, 155.93 MB, 194.91 MB, 229.17 MB, 265.52 MB, 296.08 MB, 334.73 MB, 350.70 MB, 388.37 MB, 406.23 MB, 438.94 MB, 470.84 MB, 506.38 MB, 529.30 MB, 557.37 MB, 592.28 MB, 624.67 MB, 661.28 MB, 688.48 MB, 720.17 MB, 754.94 MB, 796.02 MB, 827.47 MB, 847.64 MB, 877.79 MB, 906.30 MB, 935.29 MB, 964.11 MB, 993.82 MB, 1.01 GB, 1.05 GB, 1.09 GB, 1.13 GB, 1.17 GB, 1.19 GB, 1.21 GB, 1.23 GB, 1.26 GB, 1.29 GB - FooterProcessingTime: (Avg: 15.596ms ; Min: 3.141ms ; Max: 331.629ms ; Number of samples: 13776) - AverageHdfsReadThreadConcurrency: 0.09 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.32 GB (1416036745) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 41.33K (41328) - DecompressionTime: 318.354ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 13.78K (13776) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 17.49 MB (18344539) - PerReadThreadRawHdfsThroughput: 339.74 MB/sec - RemoteScanRanges: 123.98K (123984) - RowBatchQueueGetWaitTime: 20s116ms - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 188.29M (188286995) - RowsReturned: 188.29M (188286995) - RowsReturnedRate: 9.10 M/sec - ScanRangesComplete: 13.78K (13776) - ScannerThreadsInvoluntaryContextSwitches: 1.24K (1242) - ScannerThreadsTotalWallClockTime: 8m44s - MaterializeTupleTime(*): 3s050ms - ScannerThreadsSysTime: 1s172ms - ScannerThreadsUserTime: 6s183ms - ScannerThreadsVoluntaryContextSwitches: 101.14K (101138) - TotalRawHdfsReadTime(*): 3s974ms - TotalReadThroughput: 59.93 MB/sec Instance 564aad669bd61da1:2921b0400000001 (host=d2406.foo.com:22000):(Total: 21s648ms, non-child: 2.561ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:13459/1.50 GB MemoryUsage(500.000ms): 916.29 KB, 664.23 KB, 4.76 MB, 768.27 KB, 5.45 MB, 712.28 KB, 640.27 KB, 932.24 KB, 916.29 KB, 912.29 KB, 9.12 MB, 5.14 MB, 1.15 MB, 976.28 KB, 600.23 KB, 656.29 KB, 664.23 KB, 960.26 KB, 5.03 MB, 1.04 MB, 4.81 MB, 1.01 MB, 9.94 MB, 864.30 KB, 5.77 MB, 976.28 KB, 1.21 MB, 736.24 KB, 1.06 MB, 632.26 KB, 696.26 KB, 888.26 KB, 948.25 KB, 952.26 KB, 9.60 MB, 876.25 KB, 4.81 MB, 616.25 KB, 700.26 KB, 956.26 KB, 836.27 KB, 908.28 KB, 780.28 KB ThreadUsage(500.000ms): 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 25.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 19.26 MB (20196207) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 19.26 MB (20196207) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 206.226us - TotalStorageWaitTime: 7m27s - TotalThreadsInvoluntaryContextSwitches: 1.72K (1718) - TotalThreadsTotalWallClockTime: 8m58s - TotalThreadsSysTime: 1s476ms - TotalThreadsUserTime: 8s368ms - TotalThreadsVoluntaryContextSwitches: 250.37K (250367) Fragment Instance Lifecycle Timings: - ExecTime: 353.212us - ExecTreeExecTime: 22.022us - OpenTime: 21s587ms - ExecTreeOpenTime: 21s554ms - PrepareTime: 60.531ms - ExecTreePrepareTime: 36.868ms DataStreamSender (dst_id=2):(Total: 280.642us, non-child: 280.642us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 1.22 MB/sec - OverallThroughput: 86.99 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 8.856us - TransmitDataRPCTime: 19.527us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 54.123ms, non-child: 54.123ms, % non-child: 100.00%) - CodegenTime: 2.357ms - CompileTime: 11.120ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 21.272ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 21.110ms AGGREGATION_NODE (id=1):(Total: 21s591ms, non-child: 1s289ms, % non-child: 5.97%) ExecOption: Codegen Enabled - BuildTime: 1s168ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 176.25 KB (180480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 20s301ms, non-child: 20s301ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:13459/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 13459 out of 13459 Hdfs Read Thread Concurrency Bucket: 0:83.72% 1:13.95% 2:2.326% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:26918 BytesRead(500.000ms): 18.95 MB, 52.07 MB, 73.03 MB, 99.23 MB, 116.71 MB, 150.83 MB, 167.72 MB, 196.28 MB, 243.48 MB, 263.42 MB, 310.00 MB, 339.15 MB, 376.57 MB, 401.38 MB, 431.85 MB, 464.58 MB, 513.26 MB, 565.71 MB, 595.35 MB, 616.61 MB, 659.81 MB, 689.94 MB, 730.16 MB, 759.37 MB, 790.98 MB, 817.45 MB, 846.05 MB, 879.29 MB, 914.63 MB, 941.34 MB, 970.70 MB, 1006.45 MB, 1.01 GB, 1.04 GB, 1.07 GB, 1.09 GB, 1.12 GB, 1.15 GB, 1.17 GB, 1.21 GB, 1.23 GB, 1.25 GB, 1.30 GB - FooterProcessingTime: (Avg: 15.594ms ; Min: 1.081ms ; Max: 337.481ms ; Number of samples: 13459) - AverageHdfsReadThreadConcurrency: 0.19 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.31 GB (1409360851) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 40.38K (40377) - DecompressionTime: 327.907ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 13.46K (13459) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 19.23 MB (20159163) - PerReadThreadRawHdfsThroughput: 349.35 MB/sec - RemoteScanRanges: 121.13K (121131) - RowBatchQueueGetWaitTime: 19s696ms - RowBatchQueuePutWaitTime: 1.303ms - RowsRead: 190.13M (190134081) - RowsReturned: 190.13M (190134081) - RowsReturnedRate: 9.37 M/sec - ScanRangesComplete: 13.46K (13459) - ScannerThreadsInvoluntaryContextSwitches: 1.32K (1323) - ScannerThreadsTotalWallClockTime: 8m36s - MaterializeTupleTime(*): 3s093ms - ScannerThreadsSysTime: 1s133ms - ScannerThreadsUserTime: 6s308ms - ScannerThreadsVoluntaryContextSwitches: 98.69K (98691) - TotalRawHdfsReadTime(*): 3s847ms - TotalReadThroughput: 61.86 MB/sec Instance 564aad669bd61da1:2921b0400000006 (host=d2408.foo.com:22000):(Total: 21s200ms, non-child: 2.558ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:13092/1.50 GB MemoryUsage(500.000ms): 812.25 KB, 4.89 MB, 772.27 KB, 5.19 MB, 892.26 KB, 812.25 KB, 696.26 KB, 664.30 KB, 5.05 MB, 4.91 MB, 10.00 MB, 728.30 KB, 1.13 MB, 796.23 KB, 792.30 KB, 13.38 MB, 632.26 KB, 5.63 MB, 5.18 MB, 5.08 MB, 748.25 KB, 936.32 KB, 864.24 KB, 608.24 KB, 9.79 MB, 5.00 MB, 948.25 KB, 872.25 KB, 640.27 KB, 9.65 MB, 616.25 KB, 9.56 MB, 640.27 KB, 876.25 KB, 5.14 MB, 624.25 KB, 696.26 KB, 656.29 KB, 632.26 KB, 5.48 MB, 5.40 MB, 616.25 KB ThreadUsage(500.000ms): 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 25.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 20.36 MB (21345795) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 20.36 MB (21345795) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 143.018us - TotalStorageWaitTime: 7m18s - TotalThreadsInvoluntaryContextSwitches: 1.78K (1776) - TotalThreadsTotalWallClockTime: 8m46s - TotalThreadsSysTime: 1s255ms - TotalThreadsUserTime: 8s474ms - TotalThreadsVoluntaryContextSwitches: 243.12K (243122) Fragment Instance Lifecycle Timings: - ExecTime: 241.900us - ExecTreeExecTime: 5.035us - OpenTime: 21s140ms - ExecTreeOpenTime: 21s106ms - PrepareTime: 59.112ms - ExecTreePrepareTime: 34.929ms DataStreamSender (dst_id=2):(Total: 304.351us, non-child: 304.351us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 1.03 MB/sec - OverallThroughput: 80.22 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 7.893us - TransmitDataRPCTime: 23.067us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 55.383ms, non-child: 55.383ms, % non-child: 100.00%) - CodegenTime: 2.286ms - CompileTime: 11.238ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 21.699ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 21.832ms AGGREGATION_NODE (id=1):(Total: 21s141ms, non-child: 1s267ms, % non-child: 6.00%) ExecOption: Codegen Enabled - BuildTime: 1s152ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 176.25 KB (180480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 19s874ms, non-child: 19s874ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:13092/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 13092 out of 13092 Hdfs Read Thread Concurrency Bucket: 0:80.95% 1:16.67% 2:2.381% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:26184 BytesRead(500.000ms): 10.56 MB, 37.25 MB, 66.20 MB, 107.76 MB, 121.50 MB, 151.03 MB, 189.91 MB, 221.18 MB, 263.01 MB, 294.45 MB, 332.41 MB, 354.06 MB, 386.91 MB, 413.94 MB, 440.46 MB, 462.55 MB, 515.71 MB, 556.45 MB, 590.66 MB, 639.19 MB, 674.24 MB, 689.62 MB, 728.64 MB, 762.44 MB, 797.56 MB, 827.40 MB, 854.54 MB, 891.06 MB, 918.91 MB, 953.74 MB, 977.22 MB, 1015.97 MB, 1.03 GB, 1.05 GB, 1.08 GB, 1.11 GB, 1.13 GB, 1.17 GB, 1.18 GB, 1.21 GB, 1.23 GB, 1.28 GB - FooterProcessingTime: (Avg: 15.762ms ; Min: 2.268ms ; Max: 331.764ms ; Number of samples: 13092) - AverageHdfsReadThreadConcurrency: 0.21 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.31 GB (1406848689) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 39.28K (39276) - DecompressionTime: 335.421ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 13.09K (13092) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 20.32 MB (21303640) - PerReadThreadRawHdfsThroughput: 346.12 MB/sec - RemoteScanRanges: 117.83K (117828) - RowBatchQueueGetWaitTime: 19s306ms - RowBatchQueuePutWaitTime: 4.782ms - RowsRead: 192.78M (192782304) - RowsReturned: 192.78M (192782304) - RowsReturnedRate: 9.70 M/sec - ScanRangesComplete: 13.09K (13092) - ScannerThreadsInvoluntaryContextSwitches: 1.35K (1353) - ScannerThreadsTotalWallClockTime: 8m25s - MaterializeTupleTime(*): 3s163ms - ScannerThreadsSysTime: 985.996ms - ScannerThreadsUserTime: 6s405ms - ScannerThreadsVoluntaryContextSwitches: 96.83K (96830) - TotalRawHdfsReadTime(*): 3s876ms - TotalReadThroughput: 62.37 MB/sec Instance 564aad669bd61da1:2921b0400000009 (host=d2402.foo.com:22000):(Total: 21s197ms, non-child: 3.293ms, % non-child: 0.02%) Hdfs split stats (:<# splits>/): -1:13203/1.50 GB MemoryUsage(500.000ms): 832.27 KB, 600.23 KB, 5.72 MB, 5.29 MB, 5.24 MB, 848.29 KB, 624.25 KB, 5.12 MB, 708.27 KB, 772.27 KB, 1.19 MB, 904.27 KB, 4.91 MB, 5.30 MB, 632.26 KB, 1.07 MB, 824.25 KB, 5.46 MB, 824.26 KB, 988.29 KB, 9.07 MB, 896.26 KB, 1.14 MB, 964.26 KB, 864.30 KB, 1.14 MB, 708.27 KB, 896.26 KB, 1.17 MB, 836.27 KB, 688.25 KB, 972.28 KB, 1.26 MB, 956.26 KB, 868.24 KB, 1.09 MB, 5.24 MB, 624.25 KB, 704.27 KB, 904.28 KB, 744.25 KB, 976.29 KB ThreadUsage(500.000ms): 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 25.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 19.07 MB (19995440) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 19.08 MB (20007728) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 233.198us - TotalStorageWaitTime: 7m24s - TotalThreadsInvoluntaryContextSwitches: 1.62K (1625) - TotalThreadsTotalWallClockTime: 8m46s - TotalThreadsSysTime: 1s400ms - TotalThreadsUserTime: 8s322ms - TotalThreadsVoluntaryContextSwitches: 245.47K (245467) Fragment Instance Lifecycle Timings: - ExecTime: 433.321us - ExecTreeExecTime: 10.168us - OpenTime: 21s136ms - ExecTreeOpenTime: 21s102ms - PrepareTime: 60.279ms - ExecTreePrepareTime: 36.383ms DataStreamSender (dst_id=2):(Total: 318.527us, non-child: 318.527us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 739.55 KB/sec - OverallThroughput: 76.65 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 8.907us - TransmitDataRPCTime: 33.012us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 54.430ms, non-child: 54.430ms, % non-child: 100.00%) - CodegenTime: 2.318ms - CompileTime: 11.510ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 21.640ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 20.717ms AGGREGATION_NODE (id=1):(Total: 21s139ms, non-child: 1s269ms, % non-child: 6.00%) ExecOption: Codegen Enabled - BuildTime: 1s155ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 176.25 KB (180480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 19s869ms, non-child: 19s869ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:13203/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 13203 out of 13203 Hdfs Read Thread Concurrency Bucket: 0:90.48% 1:7.143% 2:0% 3:2.381% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:26406 BytesRead(500.000ms): 7.63 MB, 34.98 MB, 65.02 MB, 91.37 MB, 99.18 MB, 141.54 MB, 169.89 MB, 205.75 MB, 254.35 MB, 292.41 MB, 312.33 MB, 352.28 MB, 388.50 MB, 439.16 MB, 483.39 MB, 497.93 MB, 527.65 MB, 569.36 MB, 594.13 MB, 621.12 MB, 649.66 MB, 684.91 MB, 703.65 MB, 731.65 MB, 775.37 MB, 807.26 MB, 841.16 MB, 869.79 MB, 900.52 MB, 932.84 MB, 962.95 MB, 990.84 MB, 1.01 GB, 1.04 GB, 1.07 GB, 1.10 GB, 1.13 GB, 1.16 GB, 1.18 GB, 1.21 GB, 1.24 GB, 1.27 GB - FooterProcessingTime: (Avg: 15.947ms ; Min: 2.401ms ; Max: 334.065ms ; Number of samples: 13203) - AverageHdfsReadThreadConcurrency: 0.14 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.31 GB (1406115575) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 39.61K (39609) - DecompressionTime: 328.136ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 13.20K (13203) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 19.03 MB (19953285) - PerReadThreadRawHdfsThroughput: 339.87 MB/sec - RemoteScanRanges: 118.83K (118827) - RowBatchQueueGetWaitTime: 19s278ms - RowBatchQueuePutWaitTime: 3.003ms - RowsRead: 191.68M (191678805) - RowsReturned: 191.68M (191678805) - RowsReturnedRate: 9.65 M/sec - ScanRangesComplete: 13.20K (13203) - ScannerThreadsInvoluntaryContextSwitches: 1.20K (1202) - ScannerThreadsTotalWallClockTime: 8m25s - MaterializeTupleTime(*): 3s161ms - ScannerThreadsSysTime: 1s087ms - ScannerThreadsUserTime: 6s270ms - ScannerThreadsVoluntaryContextSwitches: 97.62K (97617) - TotalRawHdfsReadTime(*): 3s945ms - TotalReadThroughput: 61.80 MB/sec Instance 564aad669bd61da1:2921b0400000005 (host=d2404.foo.com:22000):(Total: 19s816ms, non-child: 2.661ms, % non-child: 0.01%) Hdfs split stats (:<# splits>/): -1:12171/1.50 GB MemoryUsage(500.000ms): 5.01 MB, 5.48 MB, 808.25 KB, 8.96 MB, 4.74 MB, 1.14 MB, 9.12 MB, 892.26 KB, 896.26 KB, 5.14 MB, 704.27 KB, 640.27 KB, 832.26 KB, 9.27 MB, 892.26 KB, 840.28 KB, 1.00 MB, 1.30 MB, 1.07 MB, 4.90 MB, 608.24 KB, 9.60 MB, 568.26 KB, 5.21 MB, 812.25 KB, 1020.26 KB, 1.17 MB, 1000.24 KB, 1.08 MB, 624.25 KB, 844.28 KB, 560.25 KB, 5.10 MB, 4.98 MB, 792.22 KB, 616.25 KB, 860.30 KB, 736.31 KB, 5.43 MB ThreadUsage(500.000ms): 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25 - AverageThreadTokens: 25.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 23.82 MB (24980308) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 23.82 MB (24980308) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 291.693us - TotalStorageWaitTime: 6m49s - TotalThreadsInvoluntaryContextSwitches: 1.71K (1708) - TotalThreadsTotalWallClockTime: 8m11s - TotalThreadsSysTime: 1s430ms - TotalThreadsUserTime: 8s295ms - TotalThreadsVoluntaryContextSwitches: 239.60K (239605) Fragment Instance Lifecycle Timings: - ExecTime: 466.388us - ExecTreeExecTime: 6.388us - OpenTime: 19s756ms - ExecTreeOpenTime: 19s722ms - PrepareTime: 58.861ms - ExecTreePrepareTime: 34.012ms DataStreamSender (dst_id=2):(Total: 373.061us, non-child: 373.061us, % non-child: 100.00%) - BytesSent: 25.00 B (25) - NetworkThroughput(*): 667.01 KB/sec - OverallThroughput: 65.44 KB/sec - PeakMemoryUsage: 7.52 KB (7704) - RowsReturned: 1 (1) - SerializeBatchTime: 10.398us - TransmitDataRPCTime: 36.602us - UncompressedRowBatchSize: 25.00 B (25) CodeGen:(Total: 56.144ms, non-child: 56.144ms, % non-child: 100.00%) - CodegenTime: 2.522ms - CompileTime: 11.168ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.96 MB (2050188) - NumFunctions: 23 (23) - NumInstructions: 384 (384) - OptimizationTime: 22.062ms - PeakMemoryUsage: 192.00 KB (196608) - PrepareTime: 22.232ms AGGREGATION_NODE (id=1):(Total: 19s756ms, non-child: 1s320ms, % non-child: 6.69%) ExecOption: Codegen Enabled - BuildTime: 1s201ms - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 176.25 KB (180480) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 18s435ms, non-child: 18s435ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): -1:12171/1.50 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 12171 out of 12171 Hdfs Read Thread Concurrency Bucket: 0:84.62% 1:10.26% 2:5.128% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0% 28:0% File Formats: PARQUET/SNAPPY:24342 BytesRead(500.000ms): 24.53 MB, 52.71 MB, 91.28 MB, 135.44 MB, 155.44 MB, 197.09 MB, 247.75 MB, 293.75 MB, 332.13 MB, 362.04 MB, 391.08 MB, 432.10 MB, 458.19 MB, 487.41 MB, 514.05 MB, 537.41 MB, 566.81 MB, 595.77 MB, 629.13 MB, 662.83 MB, 698.33 MB, 731.83 MB, 765.36 MB, 791.46 MB, 831.82 MB, 860.70 MB, 897.33 MB, 923.40 MB, 963.29 MB, 990.78 MB, 1007.93 MB, 1.00 GB, 1.04 GB, 1.09 GB, 1.12 GB, 1.18 GB, 1.21 GB, 1.25 GB, 1.28 GB - FooterProcessingTime: (Avg: 15.846ms ; Min: 1.866ms ; Max: 342.764ms ; Number of samples: 12171) - AverageHdfsReadThreadConcurrency: 0.21 - AverageScannerThreadConcurrency: 24.00 - BytesRead: 1.30 GB (1392972079) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 36.51K (36513) - DecompressionTime: 343.416ms - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 1 (1) - NumRowGroups: 12.17K (12171) - NumScannerThreadsStarted: 24 (24) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 23.78 MB (24938153) - PerReadThreadRawHdfsThroughput: 330.71 MB/sec - RemoteScanRanges: 109.54K (109539) - RowBatchQueueGetWaitTime: 17s841ms - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 197.92M (197918774) - RowsReturned: 197.92M (197918774) - RowsReturnedRate: 10.74 M/sec - ScanRangesComplete: 12.17K (12171) - ScannerThreadsInvoluntaryContextSwitches: 1.33K (1328) - ScannerThreadsTotalWallClockTime: 7m52s - MaterializeTupleTime(*): 3s231ms - ScannerThreadsSysTime: 1s123ms - ScannerThreadsUserTime: 6s183ms - ScannerThreadsVoluntaryContextSwitches: 89.41K (89406) - TotalRawHdfsReadTime(*): 4s016ms - TotalReadThroughput: 67.26 MB/sec