Query (id=ca458fe6f5d43bc9:cdd79cb300000000): DEBUG MODE WARNING: Query profile created while running a DEBUG build of Impala. Use RELEASE builds to measure query performance. Summary: Session ID: 96444fb092a3915a:9787fcca608eca9f Session Type: BEESWAX Start Time: 2018-10-19 10:40:58.053280000 End Time: 2018-10-19 10:40:58.422397000 Query Type: QUERY Query State: EXCEPTION Query Status: Memory limit exceeded: ParquetColumnReader::InitDictionary() failed to allocate 3067422 bytes for dictionary. HDFS_SCAN_NODE (id=0) could not allocate 2.93 MB without exceeding limit. Error occurred on backend mikeb-ub162:22001 by fragment ca458fe6f5d43bc9:cdd79cb300000001 Memory left in process limit: 11.83 GB Memory left in query limit: 1.38 MB Query(ca458fe6f5d43bc9:cdd79cb300000000): Limit=24.00 MB Reservation=0 ReservationLimit=0 OtherMemory=22.62 MB Total=22.62 MB Peak=22.85 MB Fragment ca458fe6f5d43bc9:cdd79cb300000001: Reservation=0 OtherMemory=22.62 MB Total=22.62 MB Peak=22.85 MB AGGREGATION_NODE (id=1): Total=20.10 KB Peak=20.10 KB Exprs: Total=4.00 KB Peak=4.00 KB HDFS_SCAN_NODE (id=0): Total=22.58 MB Peak=22.81 MB KrpcDataStreamSender (dst_id=2): Total=8.00 B Peak=8.00 B CodeGen: Total=14.38 KB Peak=932.00 KB Impala Version: impalad version 2.13.0-SNAPSHOT DEBUG (build 8ce79612f454814ff35deb372ad18f0d7e940171) User: mikeb Connected User: mikeb Delegated User: Network Address: ::ffff:127.0.0.1:37562 Default Db: tpch_parquet Sql Statement: SELECT NDV(c_custkey) AS c_custkey, CAST(-1 as BIGINT), 8, CAST(8 as DOUBLE), NDV(c_name) AS c_name, CAST(-1 as BIGINT), MAX(length(c_name)), AVG(length(c_name)), NDV(c_address) AS c_address, CAST(-1 as BIGINT), MAX(length(c_address)), AVG(length(c_address)), NDV(c_nationkey) AS c_nationkey, CAST(-1 as BIGINT), 2, CAST(2 as DOUBLE), NDV(c_phone) AS c_phone, CAST(-1 as BIGINT), MAX(length(c_phone)), AVG(length(c_phone)), NDV(c_acctbal) AS c_acctbal, CAST(-1 as BIGINT), 16, CAST(16 as DOUBLE), NDV(c_mktsegment) AS c_mktsegment, CAST(-1 as BIGINT), MAX(length(c_mktsegment)), AVG(length(c_mktsegment)), NDV(c_comment) AS c_comment, CAST(-1 as BIGINT), MAX(length(c_comment)), AVG(length(c_comment)) FROM customer Coordinator: mikeb-ub162:22000 Query Options (set by configuration): MEM_LIMIT=25165824,MT_DOP=4 Query Options (set by configuration and planner): MEM_LIMIT=25165824,NUM_SCANNER_THREADS=1,MT_DOP=4 Plan: ---------------- Max Per-Host Resource Reservation: Memory=0B Per-Host Resource Estimates: Memory=146.00MB F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B PLAN-ROOT SINK | mem-estimate=0B mem-reservation=0B | 03:AGGREGATE [FINALIZE] | output: ndv:merge(c_custkey), ndv:merge(c_name), max:merge(length(c_name)), avg:merge(length(c_name)), ndv:merge(c_address), max:merge(length(c_address)), avg:merge(length(c_address)), ndv:merge(c_nationkey), ndv:merge(c_phone), max:merge(length(c_phone)), avg:merge(length(c_phone)), ndv:merge(c_acctbal), ndv:merge(c_mktsegment), max:merge(length(c_mktsegment)), avg:merge(length(c_mktsegment)), ndv:merge(c_comment), max:merge(length(c_comment)), avg:merge(length(c_comment)) | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB | tuple-ids=2 row-size=124B cardinality=1 | 02:EXCHANGE [UNPARTITIONED] | mem-estimate=0B mem-reservation=0B | tuple-ids=1 row-size=124B cardinality=1 | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=4 Per-Host Resources: mem-estimate=136.00MB mem-reservation=0B 01:AGGREGATE | output: ndv(c_custkey), ndv(c_name), max(length(c_name)), avg(length(c_name)), ndv(c_address), max(length(c_address)), avg(length(c_address)), ndv(c_nationkey), ndv(c_phone), max(length(c_phone)), avg(length(c_phone)), ndv(c_acctbal), ndv(c_mktsegment), max(length(c_mktsegment)), avg(length(c_mktsegment)), ndv(c_comment), max(length(c_comment)), avg(length(c_comment)) | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB | tuple-ids=1 row-size=124B cardinality=1 | 00:SCAN HDFS [tpch_parquet.customer, RANDOM] partitions=1/1 files=1 size=12.31MB stored statistics: table: rows=150000 size=12.31MB columns: all extrapolated-rows=disabled mem-estimate=24.00MB mem-reservation=0B tuple-ids=0 row-size=238B cardinality=150000 ---------------- Estimated Per-Host Mem: 153092096 Per Host Min Reservation: mikeb-ub162:22000(0) mikeb-ub162:22001(0) Request Pool: default-pool Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail ------------------------------------------------------------------------------------------------------------- 03:AGGREGATE 1 0.000ns 0.000ns 0 1 0 10.00 MB FINALIZE 02:EXCHANGE 1 0.000ns 0.000ns 0 1 0 0 UNPARTITIONED 01:AGGREGATE 1 0.000ns 0.000ns 0 1 2.97 MB 10.00 MB 00:SCAN HDFS 1 28.000ms 28.000ms 0 150.00K 22.81 MB 24.00 MB tpch_parquet.customer Errors: Memory limit exceeded: ParquetColumnReader::InitDictionary() failed to allocate 3067422 bytes for dictionary. HDFS_SCAN_NODE (id=0) could not allocate 2.93 MB without exceeding limit. Error occurred on backend mikeb-ub162:22001 by fragment ca458fe6f5d43bc9:cdd79cb300000001 Memory left in process limit: 11.83 GB Memory left in query limit: 1.38 MB Query(ca458fe6f5d43bc9:cdd79cb300000000): Limit=24.00 MB Reservation=0 ReservationLimit=0 OtherMemory=22.62 MB Total=22.62 MB Peak=22.85 MB Fragment ca458fe6f5d43bc9:cdd79cb300000001: Reservation=0 OtherMemory=22.62 MB Total=22.62 MB Peak=22.85 MB AGGREGATION_NODE (id=1): Total=20.10 KB Peak=20.10 KB Exprs: Total=4.00 KB Peak=4.00 KB HDFS_SCAN_NODE (id=0): Total=22.58 MB Peak=22.81 MB KrpcDataStreamSender (dst_id=2): Total=8.00 B Peak=8.00 B CodeGen: Total=14.38 KB Peak=932.00 KB Query Compilation: 61.277ms - Metadata of all 1 tables cached: 11.030ms (11.030ms) - Analysis finished: 43.938ms (32.908ms) - Value transfer graph computed: 44.350ms (411.177us) - Single node plan created: 48.682ms (4.332ms) - Runtime filters computed: 53.551ms (4.869ms) - Distributed plan created: 53.738ms (186.842us) - Parallel plans created: 53.755ms (16.664us) - Planning finished: 61.273ms (7.517ms) - Planning finished: 61.277ms (4.163us) Query Timeline: 372.001ms - Query submitted: 0.000ns (0.000ns) - Planning finished: 64.000ms (64.000ms) - Submit for admission: 64.000ms (0.000ns) - Completed admission: 64.000ms (0.000ns) - Ready to start on 2 backends: 68.000ms (4.000ms) - All 2 execution backends (2 fragment instances) started: 68.000ms (0.000ns) - Released admission control resources: 368.001ms (300.001ms) - Rows available: 368.001ms (0.000ns) - Unregister query: 368.001ms (0.000ns) - ComputeScanRangeAssignmentTimer: 0.000ns Frontend: ImpalaServer: - ClientFetchWaitTimer: 0.000ns - RowMaterializationTimer: 0.000ns Execution Profile ca458fe6f5d43bc9:cdd79cb300000000:(Total: 304.001ms, non-child: 0.000ns, % non-child: 0.00%) Backend startup latencies: Count: 2, min / max: 1ms / 1ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms Per Node Peak Memory Usage: mikeb-ub162:22000(0) mikeb-ub162:22001(22.85 MB) - FiltersReceived: 0 (0) - FinalizationTimer: 0.000ns - NumBackends: 2 (2) - NumFragmentInstances: 2 (2) - NumFragments: 2 (2) Averaged Fragment F01: split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:300.001ms max:300.001ms mean: 300.001ms stddev:0.000ns execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 1 Coordinator Fragment F01: Instance ca458fe6f5d43bc9:cdd79cb300000000 (host=mikeb-ub162:22000): Averaged Fragment F00:(Total: 300.001ms, non-child: 8.000ms, % non-child: 2.67%) split sizes: min: 12.31 MB, max: 12.31 MB, avg: 12.31 MB, stddev: 0 completion times: min:300.001ms max:300.001ms mean: 300.001ms stddev:0.000ns execution rates: min:41.04 MB/sec max:41.04 MB/sec mean:41.04 MB/sec stddev:0.00 /sec num instances: 1 - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 22.85 MB (23958175) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 22.85 MB (23958175) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 1 (1) - TotalThreadsTotalWallClockTime: 300.001ms - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 292.000ms - TotalThreadsVoluntaryContextSwitches: 5 (5) Buffer pool: - AllocTime: 0.000ns - CumulativeAllocationBytes: 0 - CumulativeAllocations: 0 (0) - PeakReservation: 0 - PeakUnpinnedBytes: 0 - PeakUsedReservation: 0 - ReadIoBytes: 0 - ReadIoOps: 0 (0) - ReadIoWaitTime: 0.000ns - ReservationLimit: 0 - WriteIoBytes: 0 - WriteIoOps: 0 (0) - WriteIoWaitTime: 0.000ns Fragment Instance Lifecycle Timings: - OpenTime: 300.001ms - ExecTreeOpenTime: 28.000ms - PrepareTime: 0.000ns - ExecTreePrepareTime: 0.000ns KrpcDataStreamSender (dst_id=2): - EosSent: 0 (0) - PeakMemoryUsage: 8.00 B (8) - RowsSent: 0 (0) - RpcFailure: 0 (0) - RpcRetry: 0 (0) - SerializeBatchTime: 0.000ns - TotalBytesSent: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=1):(Total: 28.000ms, non-child: 0.000ns, % non-child: 0.00%) - BuildTime: 0.000ns - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 2.97 MB (3112802) - RowsRepartitioned: 0 (0) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 28.000ms, non-child: 28.000ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.00 - BytesRead: 12.41 MB (13009813) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 0 (0) - CollectionItemsRead: 0 (0) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 8 (8) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 0 (0) - NumRowGroups: 1 (1) - NumScannerThreadsStarted: 0 (0) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 22.81 MB (23914670) - PerReadThreadRawHdfsThroughput: 1.51 GB/sec - RemoteScanRanges: 0 (0) - RowsRead: 0 (0) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 1 (1) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 0.000ns - MaterializeTupleTime(*): 0.000ns - ScannerThreadsSysTime: 0.000ns - ScannerThreadsUserTime: 0.000ns - ScannerThreadsVoluntaryContextSwitches: 0 (0) - TotalRawHdfsOpenFileTime(*): 0.000ns - TotalRawHdfsReadTime(*): 8.000ms - TotalReadThroughput: 0.00 /sec CodeGen:(Total: 264.001ms, non-child: 264.001ms, % non-child: 100.00%) - CodegenTime: 16.000ms - CompileTime: 16.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2041920) - NumFunctions: 92 (92) - NumInstructions: 1.86K (1864) - OptimizationTime: 224.000ms - PeakMemoryUsage: 932.00 KB (954368) - PrepareTime: 20.000ms Fragment F00: Instance ca458fe6f5d43bc9:cdd79cb300000001 (host=mikeb-ub162:22001):(Total: 300.001ms, non-child: 8.000ms, % non-child: 2.67%) Hdfs split stats (:<# splits>/): 0:1/12.31 MB Fragment Instance Lifecycle Event Timeline: 300.001ms - Prepare Finished: 0.000ns (0.000ns) - ExecInternal Finished: 300.001ms (300.001ms) - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 22.85 MB (23958175) - PeakReservation: 0 - PeakUsedReservation: 0 - PerHostPeakMemUsage: 22.85 MB (23958175) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 1 (1) - TotalThreadsTotalWallClockTime: 300.001ms - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 292.000ms - TotalThreadsVoluntaryContextSwitches: 5 (5) Buffer pool: - AllocTime: 0.000ns - CumulativeAllocationBytes: 0 - CumulativeAllocations: 0 (0) - PeakReservation: 0 - PeakUnpinnedBytes: 0 - PeakUsedReservation: 0 - ReadIoBytes: 0 - ReadIoOps: 0 (0) - ReadIoWaitTime: 0.000ns - ReservationLimit: 0 - WriteIoBytes: 0 - WriteIoOps: 0 (0) - WriteIoWaitTime: 0.000ns Fragment Instance Lifecycle Timings: - OpenTime: 300.001ms - ExecTreeOpenTime: 28.000ms - PrepareTime: 0.000ns - ExecTreePrepareTime: 0.000ns KrpcDataStreamSender (dst_id=2): - NetworkThroughput: 0.00 /sec (Number of samples: 0) - EosSent: 0 (0) - PeakMemoryUsage: 8.00 B (8) - RowsSent: 0 (0) - RpcFailure: 0 (0) - RpcRetry: 0 (0) - SerializeBatchTime: 0.000ns - TotalBytesSent: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=1):(Total: 28.000ms, non-child: 0.000ns, % non-child: 0.00%) ExecOption: Codegen Enabled - BuildTime: 0.000ns - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 2.97 MB (3112802) - RowsRepartitioned: 0 (0) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HDFS_SCAN_NODE (id=0):(Total: 28.000ms, non-child: 28.000ms, % non-child: 100.00%) Hdfs split stats (:<# splits>/): 0:1/12.31 MB ExecOption: PARQUET Codegen Enabled - FooterProcessingTime: (Avg: 0.000ns ; Min: 0.000ns ; Max: 0.000ns ; Number of samples: 1) - AverageHdfsReadThreadConcurrency: 0.00 - BytesRead: 12.41 MB (13009813) - BytesReadDataNodeCache: 0 - BytesReadLocal: 0 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 0 - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 0 (0) - CollectionItemsRead: 0 (0) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 8 (8) - NumDictFilteredRowGroups: 0 (0) - NumDisksAccessed: 0 (0) - NumRowGroups: 1 (1) - NumScannerThreadsStarted: 0 (0) - NumScannersWithNoReads: 0 (0) - NumStatsFilteredRowGroups: 0 (0) - PeakMemoryUsage: 22.81 MB (23914670) - PerReadThreadRawHdfsThroughput: 1.51 GB/sec - RemoteScanRanges: 0 (0) - RowsRead: 0 (0) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 1 (1) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 0.000ns - MaterializeTupleTime(*): 0.000ns - ScannerThreadsSysTime: 0.000ns - ScannerThreadsUserTime: 0.000ns - ScannerThreadsVoluntaryContextSwitches: 0 (0) - TotalRawHdfsOpenFileTime(*): 0.000ns - TotalRawHdfsReadTime(*): 8.000ms - TotalReadThroughput: 0.00 /sec CodeGen:(Total: 264.001ms, non-child: 264.001ms, % non-child: 100.00%) - CodegenTime: 16.000ms - CompileTime: 16.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.95 MB (2041920) - NumFunctions: 92 (92) - NumInstructions: 1.86K (1864) - OptimizationTime: 224.000ms - PeakMemoryUsage: 932.00 KB (954368) - PrepareTime: 20.000ms