Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 2.7.0
Description
IMPALA-3629 seems to have introduced a regression in https://github.com/apache/incubator-impala/blob/master/testdata/workloads/targeted-perf/queries/primitive_filter_bigint_non_selective.test.
Interestingly of the 6 scan primitives this is the only one which regressed.
Scan node with change
HDFS_SCAN_NODE (id=0):(Total: 1s173ms, non-child: 1s173ms, % non-child: 100.00%)
ExecOption: Expr Evaluation Codegen Enabled, PARQUET Codegen Enabled, Codegen enabled: 18 out of 18
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 8:1/253.41 MB 4:2/506.80 MB 5:2/506.78 MB 7:1/253.41 MB 3:2/506.77 MB 0:2/506.77 MB 2:2/506.79 MB 1:2/506.82 MB 10:1/253.40 MB 9:2/506.79 MB 6:1/253.39 MB
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0%
File Formats: PARQUET/SNAPPY:18
BytesRead(500.000ms): 116.63 MB, 116.63 MB, 190.83 MB
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 9.67
- BytesRead: 190.83 MB (200101412)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 190.83 MB (200101412)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 190.83 MB (200101412)
- DecompressionTime: 1s618ms
- MaxCompressedTextFileLength: 0
- NumColumns: 1 (1)
- NumDisksAccessed: 11 (11)
- NumRowGroups: 18 (18)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 141.30 MB (148162336)
- PerReadThreadRawHdfsThroughput: 1.22 GB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 124.58M (124581309)
- RowsReturned: 124.58M (124581309)
- RowsReturnedRate: 106.17 M/sec
- ScanRangesComplete: 18 (18)
- ScannerThreadsInvoluntaryContextSwitches: 1.39K (1386)
- ScannerThreadsTotalWallClockTime: 14s246ms
- MaterializeTupleTime(*): 2s804ms
- ScannerThreadsSysTime: 650.896ms
- ScannerThreadsUserTime: 5s693ms
- ScannerThreadsVoluntaryContextSwitches: 124.53K (124533)
- TotalRawHdfsReadTime(*): 152.965ms
- TotalReadThroughput: 127.22 MB/sec
Scan node without change
HDFS_SCAN_NODE (id=0):(Total: 559.157ms, non-child: 559.157ms, % non-child: 100.00%)
ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 17
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/253.39 MB 6:1/253.39 MB 10:1/253.38 MB 9:2/506.79 MB 7:2/506.79 MB 8:1/253.39 MB 3:2/506.77 MB 5:1/253.40 MB 1:3/760.22 MB 4:3/760.15 MB
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0%
File Formats: PARQUET/SNAPPY:17
BytesRead(500.000ms): 116.63 MB
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 11.00
- BytesRead: 180.22 MB (188973810)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 180.22 MB (188973810)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 180.22 MB (188973810)
- DecompressionTime: 1s011ms
- MaxCompressedTextFileLength: 0
- NumColumns: 1 (1)
- NumDisksAccessed: 10 (10)
- NumRowGroups: 17 (17)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 143.67 MB (150652704)
- PerReadThreadRawHdfsThroughput: 1.45 GB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 117.66M (117660198)
- RowsReturned: 117.66M (117660198)
- RowsReturnedRate: 210.42 M/sec
- ScanRangesComplete: 17 (17)
- ScannerThreadsInvoluntaryContextSwitches: 746 (746)
- ScannerThreadsTotalWallClockTime: 7s055ms
- MaterializeTupleTime(*): 3s150ms
- ScannerThreadsSysTime: 202.962ms
- ScannerThreadsUserTime: 4s693ms
- ScannerThreadsVoluntaryContextSwitches: 48.68K (48680)
- TotalRawHdfsReadTime(*): 121.070ms
- TotalReadThroughput: 233.25 MB/sec
Attachments
Attachments
Issue Links
- relates to
-
IMPALA-2403 Replace BlockingQueue with a concurrent version so that readers and writers don't block
- Resolved