Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.8.0
Description
From the summary
| Operator | #Hosts | Avg Time | Max Time | #Rows Est. | #Rows | Peak Mem | Est. Peak Mem | Detail |
|---|---|---|---|---|---|---|---|---|
| 03:AGGREGATE | 1 | 8.668ms | 8.668ms | 1 | 1 | 7.79 MB | -1.00 B | FINALIZE |
| 02:EXCHANGE | 1 | 4.914ms | 4.914ms | 480 | 1 | 0 | -1.00 B | UNPARTITIONED |
| 01:AGGREGATE | 480 | 1s024ms | 21s191ms | 24 | 1 | 14.04 MB | -1.00 B | |
| 00:SCAN HDFS | 480 | 949.025ms | 20s143ms | 900.74M | 18.00B | 136.28 MB | -1.00 B | tpch_3000_parquet.lineitem |
The slowest fragment in the query
Instance d1403bdbfb6c9d71:156 (host=d2403.halxg.cloudera.com:22000):(Total: 36s524ms, non-child: 15.297ms, % non-child: 0.04%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 17:1/128.00 MB 20:1/128.00 MB 21:1/128.00 MB 14:2/256.00 MB 10:1/128.00 MB 7:3/384.00 MB 22:1/128.00 MB 6:1/81.40 MB 23:1/128.00 MB 11:2/218.92 MB 8:2/256.00 MB 12:1/80.91 MB 19:1/128.00 MB 2:2/256.00 MB 18:2/256.00 MB 9:2/256.00 MB
MemoryUsage(1s000ms): 40.30 KB, 112.32 MB, 90.32 MB, 79.82 MB, 108.32 MB, 63.35 MB, 84.50 MB, 88.45 MB, 105.82 MB, 108.32 MB, 130.57 MB, 113.95 MB, 86.32 MB, 126.32 MB, 55.56 MB, 108.32 MB, 92.06 MB, 61.55 MB, 112.32 MB, 90.32 MB, 69.55 MB, 108.32 MB, 63.51 MB, 75.82 MB, 130.32 MB, 97.82 MB, 94.32 MB, 130.32 MB, 90.45 MB, 76.07 MB, 123.07 MB, 90.45 MB, 86.32 MB, 94.31 MB, 75.82 MB, 126.57 MB
ThreadUsage(1s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 144.07 MB (151072208)
- PerHostPeakMemUsage: 2.54 GB (2729446952)
- RowsProduced: 1 (1)
- TotalCpuTime: 32s934ms
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 250.604us
- TotalStorageWaitTime: 3s600ms
Fragment Instance Lifecycle Timings:
- ExecTime: 563.108us
- ExecTreeExecTime: 61.100us
- OpenTime: 36s464ms
- ExecTreeOpenTime: 35s741ms
- PrepareTime: 59.505ms
- ExecTreePrepareTime: 43.876ms
CodeGen:(Total: 759.417ms, non-child: 759.417ms, % non-child: 100.00%)
- CodegenTime: 13.868ms
- CompileTime: 104.865ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.90 MB (1994596)
- NumFunctions: 166 (166)
- NumInstructions: 3.25K (3253)
- OptimizationTime: 614.018ms
- PrepareTime: 39.635ms
DataStreamSender (dst_id=2):(Total: 135.680us, non-child: 135.680us, % non-child: 100.00%)
- BytesSent: 6.21 KB (6360)
- NetworkThroughput(*): 190.50 MB/sec
- OverallThroughput: 44.70 MB/sec
- RowsReturned: 1 (1)
- SerializeBatchTime: 85.840us
- TransmitDataRPCTime: 31.840us
- UncompressedRowBatchSize: 16.54 KB (16938)
AGGREGATION_NODE (id=1):(Total: 35s749ms, non-child: 19s337ms, % non-child: 54.09%)
ExecOption: Codegen Enabled
- BuildTime: 19s016ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 13.79 MB (14464088)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 16s411ms, non-child: 16s411ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 17:1/128.00 MB 20:1/128.00 MB 21:1/128.00 MB 14:2/256.00 MB 10:1/128.00 MB 7:3/384.00 MB 22:1/128.00 MB 6:1/81.40 MB 23:1/128.00 MB 11:2/218.92 MB 8:2/256.00 MB 12:1/80.91 MB 19:1/128.00 MB 2:2/256.00 MB 18:2/256.00 MB 9:2/256.00 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 24 out of 24
Hdfs Read Thread Concurrency Bucket: 0:94.37% 1:5.634% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0%
File Formats: PARQUET/NONE:360
BytesRead(1s000ms): 0, 121.18 MB, 183.47 MB, 283.25 MB, 373.75 MB, 429.74 MB, 500.75 MB, 586.76 MB, 686.16 MB, 751.84 MB, 873.22 MB, 935.50 MB, 1001.19 MB, 1.10 GB, 1.13 GB, 1.22 GB, 1.31 GB, 1.35 GB, 1.46 GB, 1.52 GB, 1.61 GB, 1.71 GB, 1.77 GB, 1.83 GB, 1.95 GB, 2.01 GB, 2.07 GB, 2.19 GB, 2.25 GB, 2.32 GB, 2.43 GB, 2.50 GB, 2.56 GB, 2.65 GB, 2.68 GB, 2.80 GB
- FooterProcessingTime: (Avg: 16.417ms ; Min: 1.397ms ; Max: 107.045ms ; Number of samples: 24)
- AverageHdfsReadThreadConcurrency: 0.06
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 2.80 GB (3009913373)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 2.80 GB (3008684573)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 2.80 GB (3008684573)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 15 (15)
- NumDisksAccessed: 17 (17)
- NumRowGroups: 24 (24)
- NumScannerThreadsStarted: 0 (0)
- PeakMemoryUsage: 136.03 MB (142634136)
- PerReadThreadRawHdfsThroughput: 1.38 GB/sec
- RemoteScanRanges: 12 (12)
- RowsRead: 37.05M (37052184)
- RowsReturned: 37.05M (37052184)
- RowsReturnedRate: 2.26 M/sec
- ScanRangesComplete: 24 (24)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 0.000ns
- MaterializeTupleTime(*): 12s392ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 0.000ns
- ScannerThreadsVoluntaryContextSwitches: 0 (0)
- TotalRawHdfsReadTime(*): 2s026ms
- TotalReadThroughput: 78.64 MB/sec