Query (id=ce4e7a3ce060919c:d85d571d6c5fd194) Summary Session ID: e4a52d08bd1877d:75b3325443b0e09d Session Type: HIVESERVER2 HiveServer2 Protocol Version: V6 Start Time: 2016-02-23 11:28:42.765336000 End Time: 2016-02-23 11:46:41.697301000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.5.0-cdh5.7.0 DEBUG (build 22f71f3e82151c1064bf2983dd3e9025c4cce5b3) User: jenkins Connected User: jenkins Delegated User: Network Address: 172.26.26.3:55544 Default Db: tpch_10_nested_parquet Sql Statement: /* Mem: 343 MB. Coordinator: impala-stress-cdh570-4.vpc.cloudera.com. */ select 100.00 * sum(case when p_type like 'PROMO%' then l_extendedprice * (1 - l_discount) else 0.0 end) / sum(l_extendedprice * (1 - l_discount)) as promo_revenue from customer.c_orders.o_lineitems l, part p where l_partkey = p_partkey and l_shipdate >= '1995-09-01' and l_shipdate < '1995-10-01' Coordinator: impala-stress-cdh570-4.vpc.cloudera.com:22000 Query Options (non default): ABORT_ON_ERROR=1,MEM_LIMIT=359661568,RUNTIME_FILTER_MODE=2,DISABLE_ROW_RUNTIME_FILTERING=1 Plan: ---------------- Estimated Per-Host Requirements: Memory=455.57MB VCores=2 WARNING: The following tables are missing relevant table and/or column statistics. tpch_10_nested_parquet.customer 06:AGGREGATE [FINALIZE] | output: sum:merge(CASE WHEN p_type LIKE 'PROMO%' THEN l_extendedprice * (1 - l_discount) ELSE 0.0 END), sum:merge(l_extendedprice * (1 - l_discount)) | hosts=4 per-host-mem=unavailable | tuple-ids=2 row-size=32B cardinality=1 | 05:EXCHANGE [UNPARTITIONED] | hosts=4 per-host-mem=unavailable | tuple-ids=2 row-size=32B cardinality=1 | 03:AGGREGATE | output: sum(CASE WHEN p_type LIKE 'PROMO%' THEN l_extendedprice * (1 - l_discount) ELSE 0.0 END), sum(l_extendedprice * (1 - l_discount)) | hosts=4 per-host-mem=10.00MB | tuple-ids=2 row-size=32B cardinality=1 | 02:HASH JOIN [INNER JOIN, BROADCAST] | hash predicates: l_partkey = p_partkey | runtime filters: RF000 <- p_partkey | hosts=4 per-host-mem=93.57MB | tuple-ids=0,1 row-size=85B cardinality=15000000 | |--04:EXCHANGE [BROADCAST] | | hosts=4 per-host-mem=0B | | tuple-ids=1 row-size=45B cardinality=2000000 | | | 01:SCAN HDFS [tpch_10_nested_parquet.part p, RANDOM] | partitions=1/1 files=4 size=61.74MB | table stats: 2000000 rows total | column stats: all | hosts=4 per-host-mem=48.00MB | tuple-ids=1 row-size=45B cardinality=2000000 | 00:SCAN HDFS [tpch_10_nested_parquet.customer.c_orders.o_lineitems l, RANDOM] partitions=1/1 files=31 size=5.68GB predicates: l_shipdate >= '1995-09-01', l_shipdate < '1995-10-01' runtime filters: RF000 -> l_partkey table stats: 1500000 rows total column stats: all hosts=4 per-host-mem=352.00MB tuple-ids=0 row-size=40B cardinality=15000000 ---------------- Estimated Per-Host Mem: 477703193 Estimated Per-Host VCores: 2 Tables Missing Stats: tpch_10_nested_parquet.customer Request Pool: default-pool ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail ------------------------------------------------------------------------------------------------------------------------------ 06:AGGREGATE 1 1m22s 1m22s 1 1 28.00 KB -1.00 B FINALIZE 05:EXCHANGE 1 136.619us 136.619us 4 1 0 -1.00 B UNPARTITIONED 03:AGGREGATE 4 42s772ms 1m3s 4 1 4.15 MB 10.00 MB 02:HASH JOIN 4 51s719ms 1m3s 749.11K 15.00M 205.09 MB 93.57 MB INNER JOIN, BROADCAST |--04:EXCHANGE 4 10s187ms 14s619ms 2.00M 2.00M 0 0 BROADCAST | 01:SCAN HDFS 4 2m9s 4m50s 2.00M 2.00M 7.51 MB 48.00 MB tpch_10_nested_parquet.part p 00:SCAN HDFS 4 6m35s 9m12s 749.22K 15.00M 35.09 MB 352.00 MB tpch_10_nested_parquet.cust... Planner Timeline Analysis finished: 44631004 Equivalence classes computed: 256376522 Single node plan created: 1520973782 Runtime filters computed: 1521038772 Distributed plan created: 1725399543 Lineage info computed: 1726776615 Planning finished: 3581633008 Query Timeline Start execution: 36838921 Planning finished: 6119633899 Ready to start 8 remote fragments: 95230897111 All 8 remote fragments started: 141073994267 Rows available: 1078679081648 First row fetched: 1078749700192 Unregister query: 1078927445660 ImpalaServer - ClientFetchWaitTimer: 241704958 - InactiveTotalTime: 0 - RowMaterializationTimer: 29165 - TotalTime: 0 Execution Profile ce4e7a3ce060919c:d85d571d6c5fd194 Number of filters: 1 Filter routing table: ID Src. Node Dst. Node Destinations Pending Broadcast Partition filter First arrived Completed -------------------------------------------------------------------------------------------------------- 0 2 0 4 1 true false N/A N/A Fragment start latencies: Count: 8, 25th %-ile: 228ms, 50th %-ile: 400ms, 75th %-ile: 437ms, 90th %-ile: 669ms, 95th %-ile: 1s554ms, 99.9th %-ile: 1s554ms Final filter table: ID Src. Node Dst. Node Destinations Pending Broadcast Partition filter First arrived Completed -------------------------------------------------------------------------------------------------------- 0 2 0 4 1 true false N/A N/A Per Node Peak Memory Usage: impala-stress-cdh570-3.vpc.cloudera.com:22000(236.18 MB) impala-stress-cdh570-2.vpc.cloudera.com:22000(237.38 MB) impala-stress-cdh570-5.vpc.cloudera.com:22000(237.44 MB) impala-stress-cdh570-4.vpc.cloudera.com:22000(237.39 MB) - FiltersReceived: 0 - FinalizationTimer: 0 - InactiveTotalTime: 0 - TotalTime: 1071939136274 Coordinator Fragment F02 - AverageThreadTokens: 1.0 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 36880 - PerHostPeakMemUsage: 0 - PrepareTime: 82783006978 - RowsProduced: 1 - TotalCpuTime: 166343357639 - TotalNetworkReceiveTime: 906096952384 - TotalNetworkSendTime: 0 - TotalStorageWaitTime: 0 - TotalTime: 989278873279 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 64 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 254803968 - PeakMemoryUsage: 201326592 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 CodeGen - CodegenTime: 2304013 - CompileTime: 849579962 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1939944 - OptimizationTime: 29920764164 - PrepareTime: 81198835959 - TotalTime: 112404491773 AGGREGATION_NODE (id=6) ExecOption: Codegen Enabled - BuildTime: 4910 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 28672 - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 988880393752 EXCHANGE_NODE (id=5) - BytesReceived: 168 - ConvertRowBatchTime: 25230 - DeserializeRowBatchTimer: 92580 - FirstBatchArrivalWaitTime: 655008805245 - InactiveTotalTime: 906096894421 - PeakMemoryUsage: 0 - RowsReturned: 4 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 906097031040 Averaged Fragment F00 split sizes: min: 1.40 GB, max: 1.44 GB, avg: 1.42 GB, stddev: 16.90 MB completion times: min:11m28s max:15m39s mean: 12m58s stddev:1m35s execution rates: min:1.56 MB/sec max:2.09 MB/sec mean:1.89 MB/sec stddev:210.94 KB/sec num instances: 4 - AverageThreadTokens: 1.5224912418648064 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 248612092 - PerHostPeakMemUsage: 248615164 - PrepareTime: 43072903673 - RowsProduced: 1 - TotalCpuTime: 1032407994205 - TotalNetworkReceiveTime: 119663323327 - TotalNetworkSendTime: 508421 - TotalStorageWaitTime: 22579141551 - TotalTime: 631021892383 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 64 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 254803968 - PeakMemoryUsage: 201326592 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 CodeGen - CodegenTime: 249154700 - CompileTime: 12038001192 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1939944 - OptimizationTime: 133128034356 - PrepareTime: 40359889748 - TotalTime: 185967760238 DataStreamSender (dst_id=5) - BytesSent: 42 - InactiveTotalTime: 0 - NetworkThroughput(*): 107220 - OverallThroughput: 416983 - PeakMemoryUsage: 3272 - SerializeBatchTime: 20556 - ThriftTransmitTime(*): 437031 - TotalTime: 101214 - UncompressedRowBatchSize: 48 AGGREGATION_NODE (id=3) - BuildTime: 1435357637 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 4352552 - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 618357991651 HASH_JOIN_NODE (id=2) - BuildPartitionTime: 24128411321 - BuildRows: 1999744 - BuildRowsPartitioned: 1999744 - BuildTime: 16781396551 - GetNewBlockTime: 69528869 - HashBuckets: 4194304 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 6 - LocalTime: 51719026496 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 215050792 - PinTime: 0 - ProbeRows: 187305 - ProbeRowsPartitioned: 0 - ProbeTime: 105864007 - RowsReturned: 187278 - RowsReturnedRate: 327 - SpilledPartitions: 0 - TotalTime: 575585212706 - UnpinTime: 18799 EXCHANGE_NODE (id=4) - BytesReceived: 49940065 - ConvertRowBatchTime: 8157513003 - DeserializeRowBatchTimer: 18822242657 - FirstBatchArrivalWaitTime: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 0 - RowsReturned: 1999744 - RowsReturnedRate: 20073 - SendersBlockedTimer: 41752938111 - SendersBlockedTotalTimer(*): 87709786624 - TotalTime: 128149523322 HDFS_SCAN_NODE (id=0) - AverageHdfsReadThreadConcurrency: 0.006404179507300796 - AverageScannerThreadConcurrency: 1.0 - BytesRead: 342083313 - BytesReadDataNodeCache: 0 - BytesReadLocal: 341776113 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 341776113 - DecompressionTime: 0 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 4 - NumDisksAccessed: 2 - NumRowGroups: 15 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 36791242 - PerReadThreadRawHdfsThroughput: 153728116 - RemoteScanRanges: 3 - RowsRead: 15121517 - RowsReturned: 187305 - RowsReturnedRate: 494 - ScanRangesComplete: 15 - ScannerThreadsInvoluntaryContextSwitches: 22909 - ScannerThreadsTotalWallClockTime: 396371223882 - MaterializeTupleTime(*): 372670132176 - ScannerThreadsSysTime: 210967250 - ScannerThreadsUserTime: 32178607250 - ScannerThreadsVoluntaryContextSwitches: 259 - TotalRawHdfsReadTime(*): 2380625059 - TotalReadThroughput: 483032 - TotalTime: 395716662888 Averaged Fragment F01 split sizes: min: 12.38 MB, max: 18.50 MB, avg: 15.43 MB, stddev: 3.04 MB completion times: min:5m23s max:5m54s mean: 5m31s stddev:13s491ms execution rates: min:39.23 KB/sec max:58.35 KB/sec mean:47.52 KB/sec stddev:8.47 KB/sec num instances: 4 - AverageThreadTokens: 1.9846903658560326 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 6590456 - PerHostPeakMemUsage: 168330946 - PrepareTime: 79919806 - RowsProduced: 500000 - TotalCpuTime: 419475902891 - TotalNetworkReceiveTime: 0 - TotalNetworkSendTime: 180527344665 - TotalStorageWaitTime: 44810073276 - TotalTime: 328981975124 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 64 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 254803968 - PeakMemoryUsage: 134217728 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 DataStreamSender (dst_id=4) - BytesSent: 49945972 - InactiveTotalTime: 0 - NetworkThroughput(*): 2297027 - OverallThroughput: 1730009 - PeakMemoryUsage: 16384 - SerializeBatchTime: 3058678660 - ThriftTransmitTime(*): 203299838419 - TotalTime: 185038176571 - UncompressedRowBatchSize: 113206569 HDFS_SCAN_NODE (id=1) - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.9958096243401562 - BytesRead: 2690740 - BytesReadDataNodeCache: 0 - BytesReadLocal: 2690740 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 2690740 - DecompressionTime: 69947650 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 2 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 6565880 - PerReadThreadRawHdfsThroughput: 1210376638 - RemoteScanRanges: 0 - RowsRead: 500000 - RowsReturned: 500000 - RowsReturnedRate: 6410 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 1561 - ScannerThreadsTotalWallClockTime: 313359940035 - MaterializeTupleTime(*): 191297328260 - ScannerThreadsSysTime: 40243250 - ScannerThreadsUserTime: 399688750 - ScannerThreadsVoluntaryContextSwitches: 693 - TotalRawHdfsReadTime(*): 35639967 - TotalReadThroughput: 8930 - TotalTime: 129588362726 Fragment F00 Instance ce4e7a3ce060919c:d85d571d6c5fd199 (host=impala-stress-cdh570-4.vpc.cloudera.com:22000) Hdfs split stats (:<# splits>/): 0:16/1.43 GB - AverageThreadTokens: 1.6430644225188624 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 248914200 - PerHostPeakMemUsage: 248926488 - PrepareTime: 61171056567 - RowsProduced: 1 - TotalCpuTime: 1412630720963 - TotalNetworkReceiveTime: 42009873633 - TotalNetworkSendTime: 309451 - TotalStorageWaitTime: 39864684332 - TotalTime: 764589472329 CodeGen - CodegenTime: 10323902 - CompileTime: 9744333449 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1939944 - OptimizationTime: 164081666871 - PrepareTime: 59052474650 - TotalTime: 233405465277 DataStreamSender (dst_id=5) - BytesSent: 42 - InactiveTotalTime: 0 - NetworkThroughput(*): 167231 - OverallThroughput: 463279 - PeakMemoryUsage: 3272 - SerializeBatchTime: 20492 - ThriftTransmitTime(*): 251149 - TotalTime: 90658 - UncompressedRowBatchSize: 48 AGGREGATION_NODE (id=3) ExecOption: Codegen Enabled - BuildTime: 3704414417 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 4352552 - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 716103665445 HASH_JOIN_NODE (id=2) ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Build-Side Runtime-Filter Produced - BuildPartitionTime: 20756371547 - BuildRows: 1998976 - BuildRowsPartitioned: 1998976 - BuildTime: 18718945943 - GetNewBlockTime: 723057 - HashBuckets: 4194304 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 6 - LocalTime: 51312078329 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 215050792 - PinTime: 0 - ProbeRows: 190286 - ProbeRowsPartitioned: 0 - ProbeTime: 106185939 - RowsReturned: 190175 - RowsReturnedRate: 291 - SpilledPartitions: 0 - TotalTime: 652500276546 - UnpinTime: 18032 EXCHANGE_NODE (id=4) - BytesReceived: 49922346 - ConvertRowBatchTime: 6078485975 - DeserializeRowBatchTimer: 17934163245 - FirstBatchArrivalWaitTime: 0 - InactiveTotalTime: 41150799579 - PeakMemoryUsage: 0 - RowsReturned: 1998976 - RowsReturnedRate: 40815 - SendersBlockedTimer: 121212084814 - SendersBlockedTotalTimer(*): 214224270262 - TotalTime: 48976447279 HDFS_SCAN_NODE (id=0) ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 16 Hdfs split stats (:<# splits>/): 0:16/1.43 GB Hdfs Read Thread Concurrency Bucket: 0:99.64% 1:0.361% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:64 - AverageHdfsReadThreadConcurrency: 0.0036101083032490976 - AverageScannerThreadConcurrency: 1.0 - BytesRead: 347940110 - BytesReadDataNodeCache: 0 - BytesReadLocal: 347632910 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 347632910 - DecompressionTime: 0 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 4 - NumDisksAccessed: 2 - NumRowGroups: 16 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 36791240 - PerReadThreadRawHdfsThroughput: 103077104 - RemoteScanRanges: 3 - RowsRead: 15377324 - RowsReturned: 190286 - RowsReturnedRate: 344 - ScanRangesComplete: 16 - ScannerThreadsInvoluntaryContextSwitches: 23899 - ScannerThreadsTotalWallClockTime: 555346378956 - MaterializeTupleTime(*): 514037348820 - ScannerThreadsSysTime: 221966000 - ScannerThreadsUserTime: 32909996000 - ScannerThreadsVoluntaryContextSwitches: 359 - TotalRawHdfsReadTime(*): 3375532433 - TotalReadThroughput: 423195 - TotalTime: 552211750938 Instance ce4e7a3ce060919c:d85d571d6c5fd196 (host=impala-stress-cdh570-5.vpc.cloudera.com:22000) Hdfs split stats (:<# splits>/): 0:13/1.41 GB - AverageThreadTokens: 1.5006729475100942 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 248971544 - PerHostPeakMemUsage: 248971544 - PrepareTime: 35118127709 - RowsProduced: 1 - TotalCpuTime: 964950665150 - TotalNetworkReceiveTime: 148776324777 - TotalNetworkSendTime: 613323 - TotalStorageWaitTime: 15659811813 - TotalTime: 629626081458 CodeGen - CodegenTime: 407271020 - CompileTime: 10534265133 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1939944 - OptimizationTime: 116527219564 - PrepareTime: 31351858707 - TotalTime: 159034593310 DataStreamSender (dst_id=5) - BytesSent: 42 - InactiveTotalTime: 0 - NetworkThroughput(*): 77449 - OverallThroughput: 395968 - PeakMemoryUsage: 3272 - SerializeBatchTime: 18644 - ThriftTransmitTime(*): 542289 - TotalTime: 106069 - UncompressedRowBatchSize: 48 AGGREGATION_NODE (id=3) ExecOption: Codegen Enabled - BuildTime: 1476294769 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 4352552 - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 628828446999 HASH_JOIN_NODE (id=2) ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Build-Side Runtime-Filter Produced - BuildPartitionTime: 33351821554 - BuildRows: 2000000 - BuildRowsPartitioned: 2000000 - BuildTime: 18082838312 - GetNewBlockTime: 693615 - HashBuckets: 4194304 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 6 - LocalTime: 63798428994 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 215050792 - PinTime: 0 - ProbeRows: 184909 - ProbeRowsPartitioned: 0 - ProbeTime: 103489110 - RowsReturned: 184909 - RowsReturnedRate: 311 - SpilledPartitions: 0 - TotalTime: 594021353856 - UnpinTime: 16929 EXCHANGE_NODE (id=4) - BytesReceived: 49945972 - ConvertRowBatchTime: 10377671366 - DeserializeRowBatchTimer: 14048416802 - FirstBatchArrivalWaitTime: 0 - InactiveTotalTime: 144657454466 - PeakMemoryUsage: 0 - RowsReturned: 2000000 - RowsReturnedRate: 12556 - SendersBlockedTimer: 490876349 - SendersBlockedTotalTimer(*): 1110426173 - TotalTime: 159276733464 HDFS_SCAN_NODE (id=0) ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 13 Hdfs split stats (:<# splits>/): 0:13/1.41 GB Hdfs Read Thread Concurrency Bucket: 0:99.33% 1:0.672% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:52 - AverageHdfsReadThreadConcurrency: 0.006720430107526882 - AverageScannerThreadConcurrency: 1.0 - BytesRead: 336188355 - BytesReadDataNodeCache: 0 - BytesReadLocal: 335778755 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 335778755 - DecompressionTime: 0 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 4 - NumDisksAccessed: 2 - NumRowGroups: 13 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 36791232 - PerReadThreadRawHdfsThroughput: 171643162 - RemoteScanRanges: 4 - RowsRead: 14873186 - RowsReturned: 184909 - RowsReturnedRate: 498 - ScanRangesComplete: 13 - ScannerThreadsInvoluntaryContextSwitches: 21861 - ScannerThreadsTotalWallClockTime: 372337679058 - MaterializeTupleTime(*): 356236809142 - ScannerThreadsSysTime: 177972000 - ScannerThreadsUserTime: 32331084000 - ScannerThreadsVoluntaryContextSwitches: 216 - TotalRawHdfsReadTime(*): 1958646944 - TotalReadThroughput: 469737 - TotalTime: 370946191398 Instance ce4e7a3ce060919c:d85d571d6c5fd197 (host=impala-stress-cdh570-2.vpc.cloudera.com:22000) Hdfs split stats (:<# splits>/): 0:16/1.44 GB - AverageThreadTokens: 1.4957507082152974 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 248914200 - PerHostPeakMemUsage: 248914200 - PrepareTime: 47906738895 - RowsProduced: 1 - TotalCpuTime: 936489266800 - TotalNetworkReceiveTime: 123121668451 - TotalNetworkSendTime: 648569 - TotalStorageWaitTime: 19488797436 - TotalTime: 567964887391 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 64 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 254803968 - PeakMemoryUsage: 201326592 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 CodeGen - CodegenTime: 270556774 - CompileTime: 6687047861 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1939944 - OptimizationTime: 147392217077 - PrepareTime: 44983703571 - TotalTime: 199487017568 DataStreamSender (dst_id=5) - BytesSent: 42 - InactiveTotalTime: 0 - NetworkThroughput(*): 72634 - OverallThroughput: 386889 - PeakMemoryUsage: 3272 - SerializeBatchTime: 19806 - ThriftTransmitTime(*): 578238 - TotalTime: 108558 - UncompressedRowBatchSize: 48 AGGREGATION_NODE (id=3) ExecOption: Codegen Enabled - BuildTime: 349514487 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 4352552 - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 567069404174 HASH_JOIN_NODE (id=2) ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Build-Side Runtime-Filter Produced - BuildPartitionTime: 17413884626 - BuildRows: 2000000 - BuildRowsPartitioned: 2000000 - BuildTime: 14777304401 - GetNewBlockTime: 531505 - HashBuckets: 4194304 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 6 - LocalTime: 41312094157 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 215050792 - PinTime: 0 - ProbeRows: 189840 - ProbeRowsPartitioned: 0 - ProbeTime: 107884207 - RowsReturned: 189840 - RowsReturnedRate: 364 - SpilledPartitions: 0 - TotalTime: 520914563892 - UnpinTime: 23448 EXCHANGE_NODE (id=4) - BytesReceived: 49945972 - ConvertRowBatchTime: 5432472570 - DeserializeRowBatchTimer: 5893952354 - FirstBatchArrivalWaitTime: 0 - InactiveTotalTime: 122557481576 - PeakMemoryUsage: 0 - RowsReturned: 2000000 - RowsReturnedRate: 15534 - SendersBlockedTimer: 45308791283 - SendersBlockedTotalTimer(*): 135504450064 - TotalTime: 128749111296 HDFS_SCAN_NODE (id=0) ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 16 Hdfs split stats (:<# splits>/): 0:16/1.44 GB Hdfs Read Thread Concurrency Bucket: 0:99.29% 1:0.7143% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:64 - AverageHdfsReadThreadConcurrency: 0.007142857142857143 - AverageScannerThreadConcurrency: 1.0 - BytesRead: 347033300 - BytesReadDataNodeCache: 0 - BytesReadLocal: 346726100 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 346726100 - DecompressionTime: 0 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 4 - NumDisksAccessed: 2 - NumRowGroups: 16 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 36791232 - PerReadThreadRawHdfsThroughput: 137403909 - RemoteScanRanges: 3 - RowsRead: 15336960 - RowsReturned: 189840 - RowsReturnedRate: 541 - ScanRangesComplete: 16 - ScannerThreadsInvoluntaryContextSwitches: 22389 - ScannerThreadsTotalWallClockTime: 350184956549 - MaterializeTupleTime(*): 329244028785 - ScannerThreadsSysTime: 166974000 - ScannerThreadsUserTime: 31732175000 - ScannerThreadsVoluntaryContextSwitches: 237 - TotalRawHdfsReadTime(*): 2525643558 - TotalReadThroughput: 524351 - TotalTime: 350853358439 Instance ce4e7a3ce060919c:d85d571d6c5fd198 (host=impala-stress-cdh570-3.vpc.cloudera.com:22000) Hdfs split stats (:<# splits>/): 0:16/1.40 GB - AverageThreadTokens: 1.450476889214967 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 247648424 - PerHostPeakMemUsage: 247648424 - PrepareTime: 28095691523 - RowsProduced: 1 - TotalCpuTime: 815561323907 - TotalNetworkReceiveTime: 164745426447 - TotalNetworkSendTime: 462342 - TotalStorageWaitTime: 15303272626 - TotalTime: 561907128356 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 64 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 254803968 - PeakMemoryUsage: 201326592 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 CodeGen - CodegenTime: 308467104 - CompileTime: 21186358325 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1939944 - OptimizationTime: 104511033912 - PrepareTime: 26051522065 - TotalTime: 151943964799 DataStreamSender (dst_id=5) - BytesSent: 42 - InactiveTotalTime: 0 - NetworkThroughput(*): 111568 - OverallThroughput: 421796 - PeakMemoryUsage: 3272 - SerializeBatchTime: 23283 - ThriftTransmitTime(*): 376451 - TotalTime: 99574 - UncompressedRowBatchSize: 48 AGGREGATION_NODE (id=3) ExecOption: Codegen Enabled - BuildTime: 211206876 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 0 - PeakMemoryUsage: 4352552 - RowsRepartitioned: 0 - RowsReturned: 1 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 561430449986 HASH_JOIN_NODE (id=2) ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Build-Side Runtime-Filter Produced - BuildPartitionTime: 24991567559 - BuildRows: 2000000 - BuildRowsPartitioned: 2000000 - BuildTime: 15546497550 - GetNewBlockTime: 276167301 - HashBuckets: 4194304 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 6 - LocalTime: 50453504504 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 215050792 - PinTime: 0 - ProbeRows: 184188 - ProbeRowsPartitioned: 0 - ProbeTime: 105896775 - RowsReturned: 184188 - RowsReturnedRate: 344 - SpilledPartitions: 0 - TotalTime: 534904656533 - UnpinTime: 16787 EXCHANGE_NODE (id=4) - BytesReceived: 49945972 - ConvertRowBatchTime: 10741422102 - DeserializeRowBatchTimer: 37412438228 - FirstBatchArrivalWaitTime: 0 - InactiveTotalTime: 163483334281 - PeakMemoryUsage: 0 - RowsReturned: 2000000 - RowsReturnedRate: 11389 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 175595801251 HDFS_SCAN_NODE (id=0) ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 16 Hdfs split stats (:<# splits>/): 0:16/1.40 GB Hdfs Read Thread Concurrency Bucket: 0:99.19% 1:0.8143% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:64 - AverageHdfsReadThreadConcurrency: 0.008143322475570033 - AverageScannerThreadConcurrency: 1.0 - BytesRead: 337171488 - BytesReadDataNodeCache: 0 - BytesReadLocal: 336966688 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 336966688 - DecompressionTime: 0 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 4 - NumDisksAccessed: 2 - NumRowGroups: 16 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 36791264 - PerReadThreadRawHdfsThroughput: 202788290 - RemoteScanRanges: 2 - RowsRead: 14898600 - RowsReturned: 184188 - RowsReturnedRate: 596 - ScanRangesComplete: 16 - ScannerThreadsInvoluntaryContextSwitches: 23487 - ScannerThreadsTotalWallClockTime: 307615880967 - MaterializeTupleTime(*): 291162341958 - ScannerThreadsSysTime: 276957000 - ScannerThreadsUserTime: 31741174000 - ScannerThreadsVoluntaryContextSwitches: 225 - TotalRawHdfsReadTime(*): 1662677304 - TotalReadThroughput: 514845 - TotalTime: 308855350778 Fragment F01 Instance ce4e7a3ce060919c:d85d571d6c5fd19c (host=impala-stress-cdh570-4.vpc.cloudera.com:22000) Hdfs split stats (:<# splits>/): 0:1/18.45 MB - AverageThreadTokens: 1.9678571428571427 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 7901176 - PerHostPeakMemUsage: 164031592 - PrepareTime: 175881 - RowsProduced: 600000 - TotalCpuTime: 625235833645 - TotalNetworkReceiveTime: 0 - TotalNetworkSendTime: 5334880159 - TotalStorageWaitTime: 22740289592 - TotalTime: 354209003966 DataStreamSender (dst_id=4) - BytesSent: 58960572 - InactiveTotalTime: 0 - NetworkThroughput(*): 8652833 - OverallThroughput: 6332610 - PeakMemoryUsage: 16384 - SerializeBatchTime: 2783579064 - ThriftTransmitTime(*): 6814019355 - TotalTime: 9310626822 - UncompressedRowBatchSize: 135852868 HDFS_SCAN_NODE (id=1) ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 1 Hdfs split stats (:<# splits>/): 0:1/18.45 MB Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% File Formats: PARQUET/SNAPPY:2 - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.9963235294117647 - BytesRead: 3192386 - BytesReadDataNodeCache: 0 - BytesReadLocal: 3192386 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 3192386 - DecompressionTime: 12620237 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 2 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 7876600 - PerReadThreadRawHdfsThroughput: 1549822316 - RemoteScanRanges: 0 - RowsRead: 600000 - RowsReturned: 600000 - RowsReturnedRate: 2062 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 1859 - ScannerThreadsTotalWallClockTime: 298622340765 - MaterializeTupleTime(*): 11484435338 - ScannerThreadsSysTime: 50992000 - ScannerThreadsUserTime: 444932000 - ScannerThreadsVoluntaryContextSwitches: 844 - TotalRawHdfsReadTime(*): 2059840 - TotalReadThroughput: 11378 - TotalTime: 290839003534 Instance ce4e7a3ce060919c:d85d571d6c5fd19a (host=impala-stress-cdh570-3.vpc.cloudera.com:22000) Hdfs split stats (:<# splits>/): 0:1/18.50 MB - AverageThreadTokens: 1.993730407523511 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 7901176 - PerHostPeakMemUsage: 167700176 - PrepareTime: 1628271 - RowsProduced: 600000 - TotalCpuTime: 343457906748 - TotalNetworkReceiveTime: 0 - TotalNetworkSendTime: 254776939960 - TotalStorageWaitTime: 48992037538 - TotalTime: 324018370868 DataStreamSender (dst_id=4) - BytesSent: 61736268 - InactiveTotalTime: 0 - NetworkThroughput(*): 199259 - OverallThroughput: 237402 - PeakMemoryUsage: 16384 - SerializeBatchTime: 4005707031 - ThriftTransmitTime(*): 309827844559 - TotalTime: 260049330795 - UncompressedRowBatchSize: 135850508 HDFS_SCAN_NODE (id=1) ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 1 Hdfs split stats (:<# splits>/): 0:1/18.50 MB Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% File Formats: PARQUET/SNAPPY:2 - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.9984251968503937 - BytesRead: 3192326 - BytesReadDataNodeCache: 0 - BytesReadLocal: 3192326 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 3192326 - DecompressionTime: 128149438 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 2 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 7876600 - PerReadThreadRawHdfsThroughput: 23183178 - RemoteScanRanges: 0 - RowsRead: 600000 - RowsReturned: 600000 - RowsReturnedRate: 9501 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 2361 - ScannerThreadsTotalWallClockTime: 322675590406 - MaterializeTupleTime(*): 262758438446 - ScannerThreadsSysTime: 53991000 - ScannerThreadsUserTime: 478927000 - ScannerThreadsVoluntaryContextSwitches: 771 - TotalRawHdfsReadTime(*): 137700099 - TotalReadThroughput: 9996 - TotalTime: 63149602446 Instance ce4e7a3ce060919c:d85d571d6c5fd19b (host=impala-stress-cdh570-5.vpc.cloudera.com:22000) Hdfs split stats (:<# splits>/): 0:1/12.41 MB - AverageThreadTokens: 1.9855072463768115 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 5279736 - PerHostPeakMemUsage: 173895024 - PrepareTime: 294741104 - RowsProduced: 400000 - TotalCpuTime: 338015643073 - TotalNetworkReceiveTime: 0 - TotalNetworkSendTime: 278099496726 - TotalStorageWaitTime: 26677100666 - TotalTime: 322313809774 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 64 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 254803968 - PeakMemoryUsage: 134217728 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 DataStreamSender (dst_id=4) - BytesSent: 40196216 - InactiveTotalTime: 0 - NetworkThroughput(*): 130985 - OverallThroughput: 141304 - PeakMemoryUsage: 16384 - SerializeBatchTime: 4122978526 - ThriftTransmitTime(*): 306876099207 - TotalTime: 284465757163 - UncompressedRowBatchSize: 90565320 HDFS_SCAN_NODE (id=1) ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 1 Hdfs split stats (:<# splits>/): 0:1/12.41 MB Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% File Formats: PARQUET/SNAPPY:2 - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.9935064935064936 - BytesRead: 2189284 - BytesReadDataNodeCache: 0 - BytesReadLocal: 2189284 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 2189284 - DecompressionTime: 7973222 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 2 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 5255160 - PerReadThreadRawHdfsThroughput: 1973864341 - RemoteScanRanges: 0 - RowsRead: 400000 - RowsReturned: 400000 - RowsReturnedRate: 10951 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 1242 - ScannerThreadsTotalWallClockTime: 319310792252 - MaterializeTupleTime(*): 286161128196 - ScannerThreadsSysTime: 34994000 - ScannerThreadsUserTime: 362944000 - ScannerThreadsVoluntaryContextSwitches: 503 - TotalRawHdfsReadTime(*): 1109136 - TotalReadThroughput: 7059 - TotalTime: 36525389510 Instance ce4e7a3ce060919c:d85d571d6c5fd19d (host=impala-stress-cdh570-2.vpc.cloudera.com:22000) Hdfs split stats (:<# splits>/): 0:1/12.38 MB - AverageThreadTokens: 1.9916666666666667 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 5279736 - PerHostPeakMemUsage: 167696992 - PrepareTime: 23133969 - RowsProduced: 400000 - TotalCpuTime: 371194228100 - TotalNetworkReceiveTime: 0 - TotalNetworkSendTime: 183898061816 - TotalStorageWaitTime: 80830865309 - TotalTime: 315386715889 DataStreamSender (dst_id=4) - BytesSent: 38890832 - InactiveTotalTime: 0 - NetworkThroughput(*): 205032 - OverallThroughput: 208723 - PeakMemoryUsage: 16384 - SerializeBatchTime: 1322450019 - ThriftTransmitTime(*): 189681390558 - TotalTime: 186326991504 - UncompressedRowBatchSize: 90557580 HDFS_SCAN_NODE (id=1) ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 1 Hdfs split stats (:<# splits>/): 0:1/12.38 MB Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% File Formats: PARQUET/SNAPPY:2 - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.9949832775919732 - BytesRead: 2188964 - BytesReadDataNodeCache: 0 - BytesReadLocal: 2188964 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 2188964 - DecompressionTime: 131047706 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 2 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 5255160 - PerReadThreadRawHdfsThroughput: 1294636720 - RemoteScanRanges: 0 - RowsRead: 400000 - RowsReturned: 400000 - RowsReturnedRate: 3128 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 783 - ScannerThreadsTotalWallClockTime: 312831036720 - MaterializeTupleTime(*): 204785311060 - ScannerThreadsSysTime: 20996000 - ScannerThreadsUserTime: 311952000 - ScannerThreadsVoluntaryContextSwitches: 654 - TotalRawHdfsReadTime(*): 1690794 - TotalReadThroughput: 7288 - TotalTime: 127839455415