Query (id=25425c1b54a69ce3:35572cfb2200490): Summary: Session ID: 69426eeb8a2ecfe4:d4463b3534959f9c Session Type: BEESWAX Start Time: 2016-01-04 13:29:43.554909000 End Time: Query Type: QUERY Query State: CREATED Query Status: OK Impala Version: impalad version 2.5.0-cdh5-INTERNAL DEBUG (build 2916cc2eb503af075caf22bf8354e64430e967f9) User: mikeb Connected User: mikeb Delegated User: Network Address: ::ffff:127.0.0.1:57154 Default Db: default Sql Statement: select count(l_orderkey) from functional_parquet.lineitem_multiblock Coordinator: mikeb-desktop:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=26.00MB VCores=1 WARNING: The following tables are missing relevant table and/or column statistics. functional_parquet.lineitem_multiblock F01:PLAN FRAGMENT [UNPARTITIONED] 03:AGGREGATE [FINALIZE] | output: count:merge(l_orderkey) | hosts=3 per-host-mem=unavailable | tuple-ids=1 row-size=8B cardinality=1 | 02:EXCHANGE [UNPARTITIONED] hosts=3 per-host-mem=unavailable tuple-ids=1 row-size=8B cardinality=1 F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=02, UNPARTITIONED] 01:AGGREGATE | output: count(l_orderkey) | hosts=3 per-host-mem=10.00MB | tuple-ids=1 row-size=8B cardinality=1 | 00:SCAN HDFS [functional_parquet.lineitem_multiblock, RANDOM] partitions=1/1 files=1 size=2.64MB table stats: unavailable column stats: unavailable hosts=3 per-host-mem=16.00MB tuple-ids=0 row-size=8B cardinality=unavailable ---------------- Estimated Per-Host Mem: 27262976 Estimated Per-Host VCores: 1 Tables Missing Stats: functional_parquet.lineitem_multiblock Request Pool: default-pool Planner Timeline: 1ms - Analysis finished: 472us (472us) - Equivalence classes computed: 507us (34us) - Single node plan created: 709us (202us) - Distributed plan created: 849us (139us) - Planning finished: 1ms (252us) Query Timeline: 1s608ms - Start execution: 32us (32us) - Planning finished: 1ms (1ms) - Ready to start remote fragments: 222ms (220ms) - Remote fragments started: 690ms (467ms) - Rows available: 1s601ms (911ms) - First row fetched: 1s607ms (5ms) ImpalaServer: - ClientFetchWaitTimer: 5ms - RowMaterializationTimer: 4us Execution Profile 25425c1b54a69ce3:35572cfb2200490:(Total: 1s588ms, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 2, last: 0.459584ns, min: 0.239351ns, max: 0.459584ns, mean: 0.349467ns, stddev: 0.110116ns Per Node Peak Memory Usage: mikeb-desktop:22001(852.00 KB) mikeb-desktop:22000(456.00 KB) - FinalizationTimer: 0ns Coordinator Fragment F01:(Total: 509ms, non-child: 0ns, % non-child: 0.00%) MemoryUsage(500ms): 12.00 KB, 12.00 KB, 24.01 KB ThreadUsage(500ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 28.02 KB (28688) - PerHostPeakMemUsage: 0 - PrepareTime: 220ms - RowsProduced: 1 (1) - TotalCpuTime: 1s316ms - TotalNetworkReceiveTime: 289ms - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 20.04 GB (21513172992) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 829ms, non-child: 829ms, % non-child: 100.00%) - CodegenTime: 591us - CompileTime: 446us - LoadTime: 0ns - ModuleFileSize: 0 - OptimizationTime: 609ms - PrepareTime: 218ms AGGREGATION_NODE (id=3):(Total: 509ms, non-child: 220ms, % non-child: 43.26%) ExecOption: Codegen Enabled - BuildTime: 2us - GetResultsTime: 0ns - HTResizeTime: 0ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 16.00 KB (16384) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 1.00 /sec - SpilledPartitions: 0 (0) EXCHANGE_NODE (id=2):(Total: 289ms, non-child: 24us, % non-child: 0.01%) BytesReceived(500ms): 0, 0, 16.00 B - BytesReceived: 32.00 B (32) - ConvertRowBatchTime: 3us - DeserializeRowBatchTimer: 13us - FirstBatchArrivalWaitTime: 0ns - PeakMemoryUsage: 0 - RowsReturned: 2 (2) - RowsReturnedRate: 6.00 /sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 610ms, non-child: 0ns, % non-child: 0.00%) split sizes: min: 654.38 KB, max: 2.00 MB, avg: 1.32 MB, stddev: 696.81 KB completion times: min:675ms max:911ms mean: 793.500043ms stddev:118.312243ms execution rates: min:969.19 KB/sec max:2.19 MB/sec mean:1.57 MB/sec stddev:638.44 KB/sec num instances: 2 - AverageThreadTokens: 1.33 - PeakMemoryUsage: 648.00 KB (663552) - PerHostPeakMemUsage: 654.00 KB (669696) - PrepareTime: 332ms - RowsProduced: 1 (1) - TotalCpuTime: 1s189ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 174us - TotalStorageWaitTime: 445ms CodeGen:(Total: 847ms, non-child: 847ms, % non-child: 100.00%) - CodegenTime: 603us - CompileTime: 939us - LoadTime: 0ns - ModuleFileSize: 0 - OptimizationTime: 514ms - PrepareTime: 331ms DataStreamSender (dst_id=2):(Total: 45us, non-child: 45us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 134.57 KB/sec - OverallThroughput: 348.33 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 8us - ThriftTransmitTime(*): 150us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=1):(Total: 603ms, non-child: 332ms, % non-child: 55.15%) - BuildTime: 83us - GetResultsTime: 0ns - HTResizeTime: 0ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 48.00 KB (49152) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 2.00 /sec - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 270ms, non-child: 270ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 1.00 - BytesRead: 505.36 KB (517485) - BytesReadDataNodeCache: 0 - BytesReadLocal: 505.36 KB (517485) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 505.36 KB (517485) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 1 (1) - NumRowGroups: 100 (100) - NumScannerThreadsStarted: 1 (1) - PeakMemoryUsage: 612.00 KB (626688) - PerReadThreadRawHdfsThroughput: 60.25 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 10.00K (10000) - RowsReturned: 10.00K (10000) - RowsReturnedRate: 38.59 K/sec - ScanRangesComplete: 1 (1) - ScannerThreadsInvoluntaryContextSwitches: 216 (216) - ScannerThreadsTotalWallClockTime: 493ms - MaterializeTupleTime(*): 5ms - ScannerThreadsSysTime: 0ns - ScannerThreadsUserTime: 26ms - ScannerThreadsVoluntaryContextSwitches: 205 (205) - TotalRawHdfsReadTime(*): 8ms - TotalReadThroughput: 324.30 KB/sec BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 20.04 GB (21513172992) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns Fragment F00: Instance 25425c1b54a69ce3:35572cfb2200493 (host=mikeb-desktop:22001):(Total: 899ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:2/2.00 MB MemoryUsage(500ms): 0, 28.00 KB, 108.00 KB ThreadUsage(500ms): 1, 1, 3 - AverageThreadTokens: 1.67 - PeakMemoryUsage: 852.00 KB (872448) - PerHostPeakMemUsage: 852.00 KB (872448) - PrepareTime: 428ms - RowsProduced: 1 (1) - TotalCpuTime: 1s442ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 103us - TotalStorageWaitTime: 830ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 20.04 GB (21513172992) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns CodeGen:(Total: 870ms, non-child: 870ms, % non-child: 100.00%) - CodegenTime: 565us - CompileTime: 444us - LoadTime: 0ns - ModuleFileSize: 0 - OptimizationTime: 442ms - PrepareTime: 426ms DataStreamSender (dst_id=2):(Total: 49us, non-child: 49us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 198.62 KB/sec - OverallThroughput: 314.87 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 7us - ThriftTransmitTime(*): 78us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=1):(Total: 884ms, non-child: 428ms, % non-child: 48.46%) ExecOption: Codegen Enabled - BuildTime: 125us - GetResultsTime: 0ns - HTResizeTime: 0ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 56.00 KB (57344) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 1.00 /sec - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 455ms, non-child: 455ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:2/2.00 MB Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% File Formats: PARQUET/NONE:2 ExecOption: Codegen enabled: 0 out of 2 BytesRead(500ms): 0, 648.61 KB - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 2.00 - BytesRead: 684.31 KB (700733) - BytesReadDataNodeCache: 0 - BytesReadLocal: 684.31 KB (700733) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 684.31 KB (700733) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 2 (2) - NumRowGroups: 165 (165) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 816.00 KB (835584) - PerReadThreadRawHdfsThroughput: 52.72 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 16.50K (16500) - RowsReturned: 16.50K (16500) - RowsReturnedRate: 36.20 K/sec - ScanRangesComplete: 2 (2) - ScannerThreadsInvoluntaryContextSwitches: 199 (199) - ScannerThreadsTotalWallClockTime: 902ms - MaterializeTupleTime(*): 8ms - ScannerThreadsSysTime: 0ns - ScannerThreadsUserTime: 40ms - ScannerThreadsVoluntaryContextSwitches: 348 (348) - TotalRawHdfsReadTime(*): 12ms - TotalReadThroughput: 648.61 KB/sec Instance 25425c1b54a69ce3:35572cfb2200492 (host=mikeb-desktop:22000):(Total: 322ms, non-child: 0ns, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:1/654.38 KB MemoryUsage(500ms): 0, 28.00 KB ThreadUsage(500ms): 1, 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 444.00 KB (454656) - PerHostPeakMemUsage: 456.00 KB (466944) - PrepareTime: 236ms - RowsProduced: 1 (1) - TotalCpuTime: 935ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 245us - TotalStorageWaitTime: 61ms CodeGen:(Total: 824ms, non-child: 824ms, % non-child: 100.00%) - CodegenTime: 642us - CompileTime: 1ms - LoadTime: 0ns - ModuleFileSize: 0 - OptimizationTime: 586ms - PrepareTime: 235ms DataStreamSender (dst_id=2):(Total: 40us, non-child: 40us, % non-child: 100.00%) - BytesSent: 16.00 B (16) - NetworkThroughput(*): 70.52 KB/sec - OverallThroughput: 381.79 KB/sec - PeakMemoryUsage: 16.00 KB (16384) - SerializeBatchTime: 9us - ThriftTransmitTime(*): 221us - UncompressedRowBatchSize: 16.00 B (16) AGGREGATION_NODE (id=1):(Total: 322ms, non-child: 236ms, % non-child: 73.51%) ExecOption: Codegen Enabled - BuildTime: 42us - GetResultsTime: 0ns - HTResizeTime: 0ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 40.00 KB (40960) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 3.00 /sec - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 85ms, non-child: 85ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:1/654.38 KB Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0% 5:0% File Formats: PARQUET/NONE:1 ExecOption: Codegen enabled: 0 out of 1 BytesRead(500ms): 0 - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 0.00 - BytesRead: 326.40 KB (334237) - BytesReadDataNodeCache: 0 - BytesReadLocal: 326.40 KB (334237) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 326.40 KB (334237) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 1 (1) - NumRowGroups: 35 (35) - NumScannerThreadsStarted: 1 (1) - PeakMemoryUsage: 408.00 KB (417792) - PerReadThreadRawHdfsThroughput: 67.78 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 3.50K (3500) - RowsReturned: 3.50K (3500) - RowsReturnedRate: 40.99 K/sec - ScanRangesComplete: 1 (1) - ScannerThreadsInvoluntaryContextSwitches: 234 (234) - ScannerThreadsTotalWallClockTime: 84ms - MaterializeTupleTime(*): 2ms - ScannerThreadsSysTime: 0ns - ScannerThreadsUserTime: 12ms - ScannerThreadsVoluntaryContextSwitches: 62 (62) - TotalRawHdfsReadTime(*): 4ms - TotalReadThroughput: 0.00 /sec