Query (id=f44a3eb878bdb29e:cc59ef7f71c441b9) Summary Session ID: 844225e53b4ec00a:8c72d2c457329a98 Session Type: HIVESERVER2 HiveServer2 Protocol Version: V6 Start Time: 2016-10-14 11:00:15.998225000 End Time: Query Type: QUERY Query State: CREATED Query Status: OK Impala Version: impalad version 2.5.0-cdh5.7.1 RELEASE (build 27a4325c18c2a01c7a8097681a0eccf6d4335ea1) User: x194245 Connected User: x194245 Delegated User: Network Address: 48.24.145.193:57950 Default Db: retpocd01 Sql Statement: SELECT `v_univl_cust_dim`.`univl_cust_id` as `univl_cust_id`, `v_univl_cust_dim`.`yyyymm_key` as `yyyymm_key`, `v_univl_cust_dim`.`home_email_id` as `Home_Email_Address`, `v_univl_cust_dim`.`wrk_email_id` as `Work_Email_Address`, `v_univl_cust_dim`.`email_prefnce_ind` as `Preferred_Email_Address` FROM `retpocd01`.`v_univl_cust_dim` `v_univl_cust_dim` INNER JOIN `retpocd01`.`v_t_partcpnt_no_test_dummy_dim` `v_t_partcpnt_no_test_dummy_dim` ON ((`v_univl_cust_dim`.`yyyymm_key` = `v_t_partcpnt_no_test_dummy_dim`.`yyyymm_key`) AND (`v_univl_cust_dim`.`univl_cust_id` = `v_t_partcpnt_no_test_dummy_dim`.`univl_cust_id`)) WHERE ((`v_t_partcpnt_no_test_dummy_dim`.`dc_sub_plan_num` IS NULL) AND ((`v_univl_cust_dim`.`yyyymm_key` = 201509) AND (`v_t_partcpnt_no_test_dummy_dim`.`dc_plan_num` = '763575'))) GROUP BY `v_univl_cust_dim`.`univl_cust_id`, `v_univl_cust_dim`.`yyyymm_key`, `v_univl_cust_dim`.`home_email_id`, `v_univl_cust_dim`.`wrk_email_id`, `v_univl_cust_dim`.`email_prefnce_ind` Coordinator: paehowchq1d02.prudential.com:22000 Query Options (non default): Plan: ---------------- Estimated Per-Host Requirements: Memory=2.68GB VCores=2 WARNING: The following tables are missing relevant table and/or column statistics. retpocd01.v_t_partcpnt_no_test_dummy_dim, retpocd01.v_univl_cust_dim 07:EXCHANGE [UNPARTITIONED] | hosts=3 per-host-mem=unavailable | tuple-ids=2 row-size=57B cardinality=unavailable | 06:AGGREGATE [FINALIZE] | group by: v_univl_cust_dim.univl_cust_id, v_univl_cust_dim.yyyymm_key, v_univl_cust_dim.home_email_id, v_univl_cust_dim.wrk_email_id, v_univl_cust_dim.email_prefnce_ind | hosts=3 per-host-mem=128.00MB | tuple-ids=2 row-size=57B cardinality=unavailable | 05:EXCHANGE [HASH(v_univl_cust_dim.univl_cust_id,v_univl_cust_dim.yyyymm_key,v_univl_cust_dim.home_email_id,v_univl_cust_dim.wrk_email_id,v_univl_cust_dim.email_prefnce_ind)] | hosts=3 per-host-mem=0B | tuple-ids=2 row-size=57B cardinality=unavailable | 03:AGGREGATE [STREAMING] | group by: v_univl_cust_dim.univl_cust_id, v_univl_cust_dim.yyyymm_key, v_univl_cust_dim.home_email_id, v_univl_cust_dim.wrk_email_id, v_univl_cust_dim.email_prefnce_ind | hosts=3 per-host-mem=128.00MB | tuple-ids=2 row-size=57B cardinality=unavailable | 02:HASH JOIN [INNER JOIN, BROADCAST] | hash predicates: v_univl_cust_dim.yyyymm_key = v_t_partcpnt_no_test_dummy_dim.yyyymm_key, v_univl_cust_dim.univl_cust_id = v_t_partcpnt_no_test_dummy_dim.univl_cust_id | runtime filters: RF000 <- v_t_partcpnt_no_test_dummy_dim.yyyymm_key, RF001 <- v_t_partcpnt_no_test_dummy_dim.univl_cust_id | hosts=3 per-host-mem=2.00GB | tuple-ids=0,1 row-size=99B cardinality=unavailable | |--04:EXCHANGE [BROADCAST] | | hosts=1 per-host-mem=0B | | tuple-ids=1 row-size=42B cardinality=unavailable | | | 01:SCAN HDFS [retpocd01.v_t_partcpnt_no_test_dummy_dim v_t_partcpnt_no_test_dummy_dim, RANDOM] | partitions=1/24 files=1 size=200.05MB | predicates: (v_t_partcpnt_no_test_dummy_dim.dc_sub_plan_num IS NULL), (v_t_partcpnt_no_test_dummy_dim.dc_plan_num = '763575') | table stats: unavailable | columns missing stats: univl_cust_id, dc_sub_plan_num, dc_plan_num | hosts=1 per-host-mem=264.00MB | tuple-ids=1 row-size=42B cardinality=unavailable | 00:SCAN HDFS [retpocd01.v_univl_cust_dim v_univl_cust_dim, RANDOM] partitions=1/1 files=3 size=689.50MB predicates: (v_univl_cust_dim.yyyymm_key = 201509) runtime filters: RF000 -> v_univl_cust_dim.yyyymm_key, RF001 -> v_univl_cust_dim.univl_cust_id table stats: unavailable column stats: unavailable hosts=3 per-host-mem=440.00MB tuple-ids=0 row-size=57B cardinality=unavailable ---------------- Estimated Per-Host Mem: 2877292544 Estimated Per-Host VCores: 2 Tables Missing Stats: retpocd01.v_t_partcpnt_no_test_dummy_dim,retpocd01.v_univl_cust_dim Request Pool: root.retpocfad01 Admission result: Admitted immediately Planner Timeline Analysis finished: 5649701 Equivalence classes computed: 5982197 Single node plan created: 9493098 Runtime filters computed: 9876070 Distributed plan created: 12209010 Lineage info computed: 13047150 Planning finished: 16231864 Query Timeline Start execution: 66403 Planning finished: 18765821 Submit for admission: 19653841 Completed admission: 19771296 Ready to start 7 remote fragments: 20565681 All 7 remote fragments started: 32462485 Rows available: 1726286737 ImpalaServer - ClientFetchWaitTimer: 0 - InactiveTotalTime: 0 - RowMaterializationTimer: 0 - TotalTime: 0 Execution Profile f44a3eb878bdb29e:cc59ef7f71c441b9 Number of filters: 2 Filter routing table: ID Src. Node Tgt. Node Targets Type Partition filter ------------------------------------------------------------ 1 2 0 3 LOCAL false 0 2 0 3 LOCAL false Fragment start latencies: Count: 7, 25th %-ile: 2ms, 50th %-ile: 3ms, 75th %-ile: 4ms, 90th %-ile: 12ms, 95th %-ile: 12ms, 99.9th %-ile: 12ms Final filter table: ID Src. Node Tgt. Node Targets Type Partition filter ------------------------------------------------------------ 1 2 0 3 LOCAL false 0 2 0 3 LOCAL false - FiltersReceived: 0 - FinalizationTimer: 0 - InactiveTotalTime: 0 - TotalTime: 1705908654 Coordinator Fragment F03 - AverageThreadTokens: 0.0 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 8192 - PerHostPeakMemUsage: 0 - PrepareTime: 48896 - RowsProduced: 0 - TotalCpuTime: 0 - TotalNetworkReceiveTime: 1692984517 - TotalNetworkSendTime: 0 - TotalStorageWaitTime: 0 - TotalTime: 1693418725 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 80 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 150323855360 - PeakMemoryUsage: 524288 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 EXCHANGE_NODE (id=7) - BytesReceived: 0 - ConvertRowBatchTime: 0 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 1692977559 - InactiveTotalTime: 1692981139 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 1692986508 Averaged Fragment F02 split sizes: min: 0, max: 0, avg: 0, stddev: 0 - AverageThreadTokens: 1.0 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 2377728 - PerHostPeakMemUsage: 143510778 - PrepareTime: 256460748 - RowsProduced: 0 - TotalCpuTime: 488955917 - TotalNetworkReceiveTime: 1211394330 - TotalNetworkSendTime: 313 - TotalStorageWaitTime: 0 - TotalTime: 1469556707 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 80 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 150323855360 - PeakMemoryUsage: 524288 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 CodeGen - CodegenTime: 2004509 - CompileTime: 27414964 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 202519901 - PrepareTime: 251971820 - TotalTime: 483973647 DataStreamSender (dst_id=7) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 2048 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 12020 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=6) - BuildTime: 7837 - GetNewBlockTime: 122790 - GetResultsTime: 0 - HTResizeTime: 2967 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 2367488 - PinTime: 0 - RowsRepartitioned: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 1468562362 - UnpinTime: 937 EXCHANGE_NODE (id=5) - BytesReceived: 0 - ConvertRowBatchTime: 1228 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 1211389999 - InactiveTotalTime: 0 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 1211399270 Averaged Fragment F00 split sizes: min: 217.32 MB, max: 242.41 MB, avg: 229.83 MB, stddev: 10.24 MB - AverageThreadTokens: 1.6944444444444446 - BloomFilterBytes: 2097152 - InactiveTotalTime: 0 - PeakMemoryUsage: 139041360 - PerHostPeakMemUsage: 143510778 - PrepareTime: 261147908 - RowsProduced: 0 - TotalCpuTime: 2653160092 - TotalNetworkReceiveTime: 56660729 - TotalNetworkSendTime: 1492 - TotalStorageWaitTime: 45010314 - TotalTime: 1373810969 CodeGen - CodegenTime: 1732331 - CompileTime: 45213164 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 225419923 - PrepareTime: 254465616 - TotalTime: 527159149 DataStreamSender (dst_id=5) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 6144 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 13633 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=3) - GetNewBlockTime: 223791 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 1327104 - PinTime: 0 - ReductionFactorEstimate: 0.0 - ReductionFactorThresholdToExpand: 0.0 - RowsPassedThrough: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - StreamingTime: 2006 - TotalTime: 1371751276 - UnpinTime: 0 HASH_JOIN_NODE (id=2) - BuildPartitionTime: 2569 - BuildRows: 0 - BuildRowsPartitioned: 0 - BuildTime: 0 - GetNewBlockTime: 191410 - HashBuckets: 0 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - LocalTime: 6941293 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 9042405 - PinTime: 0 - ProbeRows: 0 - ProbeRowsPartitioned: 0 - ProbeTime: 1465 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 1113100742 - UnpinTime: 1949 EXCHANGE_NODE (id=4) - BytesReceived: 0 - ConvertRowBatchTime: 1022 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 56654809 - InactiveTotalTime: 0 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 56665135 HDFS_SCAN_NODE (id=0) - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.8888888888888888 - BytesRead: 128736734 - BytesReadDataNodeCache: 0 - BytesReadLocal: 128736734 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 128736734 - DecompressionTime: 358400396 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 5 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 137690800 - PerReadThreadRawHdfsThroughput: 739531974 - RemoteScanRanges: 0 - RowsRead: 5590131 - RowsReturned: 0 - RowsReturnedRate: 0 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 2 - ScannerThreadsTotalWallClockTime: 1050763739 - MaterializeTupleTime(*): 623003727 - ScannerThreadsSysTime: 6998333 - ScannerThreadsUserTime: 997181000 - ScannerThreadsVoluntaryContextSwitches: 7 - TotalRawHdfsReadTime(*): 174981974 - TotalReadThroughput: 100130379 - TotalTime: 1049494313 Filter 0 - InactiveTotalTime: 0 - Rows processed: 5590131 - Rows rejected: 5590131 - Rows total: 5590131 - TotalTime: 0 Filter 1 - InactiveTotalTime: 0 - Rows processed: 0 - Rows rejected: 0 - Rows total: 0 - TotalTime: 0 Averaged Fragment F01 split sizes: min: 200.05 MB, max: 200.05 MB, avg: 200.05 MB, stddev: 0 - AverageThreadTokens: 1.0 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 26516552 - PerHostPeakMemUsage: 30219336 - PrepareTime: 262373045 - RowsProduced: 0 - TotalCpuTime: 722171551 - TotalNetworkReceiveTime: 0 - TotalNetworkSendTime: 1623 - TotalStorageWaitTime: 16915604 - TotalTime: 409819215 CodeGen - CodegenTime: 0 - CompileTime: 1471491 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 181009760 - PrepareTime: 260804255 - TotalTime: 444347099 DataStreamSender (dst_id=4) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 8184 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 24995 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=1) - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.0 - BytesRead: 13557142 - BytesReadDataNodeCache: 0 - BytesReadLocal: 13557142 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 13557142 - DecompressionTime: 23086127 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 3 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 26442824 - PerReadThreadRawHdfsThroughput: 1156141021 - RemoteScanRanges: 0 - RowsRead: 1463085 - RowsReturned: 0 - RowsReturnedRate: 0 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 0 - ScannerThreadsTotalWallClockTime: 145863138 - MaterializeTupleTime(*): 103748688 - ScannerThreadsSysTime: 3999000 - ScannerThreadsUserTime: 123981000 - ScannerThreadsVoluntaryContextSwitches: 8 - TotalRawHdfsReadTime(*): 11726201 - TotalReadThroughput: 0 - TotalTime: 670389734 Fragment F02 Instance f44a3eb878bdb29e:cc59ef7f71c441bc (host=paehowchq1d03.prudential.com:22000) - AverageThreadTokens: 1.0 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 2377728 - PerHostPeakMemUsage: 144889608 - PrepareTime: 260745555 - RowsProduced: 0 - TotalCpuTime: 489132480 - TotalNetworkReceiveTime: 1204506944 - TotalNetworkSendTime: 328 - TotalStorageWaitTime: 0 - TotalTime: 1466980875 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 80 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 150323855360 - PeakMemoryUsage: 524288 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 CodeGen - CodegenTime: 1959613 - CompileTime: 26801503 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 199255015 - PrepareTime: 256089986 - TotalTime: 484415040 DataStreamSender (dst_id=7) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 2048 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 12209 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=6) ExecOption: Codegen Enabled - BuildTime: 8204 - GetNewBlockTime: 128197 - GetResultsTime: 0 - HTResizeTime: 3154 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 2367488 - PinTime: 0 - RowsRepartitioned: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 1465836281 - UnpinTime: 905 EXCHANGE_NODE (id=5) - BytesReceived: 0 - ConvertRowBatchTime: 1022 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 1204502610 - InactiveTotalTime: 1204503750 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 1204512281 Instance f44a3eb878bdb29e:cc59ef7f71c441bb (host=paehowchq1d01.prudential.com:22000) - AverageThreadTokens: 1.0 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 2377728 - PerHostPeakMemUsage: 140708960 - PrepareTime: 264492019 - RowsProduced: 0 - TotalCpuTime: 503144239 - TotalNetworkReceiveTime: 1200126963 - TotalNetworkSendTime: 300 - TotalStorageWaitTime: 0 - TotalTime: 1466341561 BlockMgr - BlockWritesOutstanding: 0 - BlocksCreated: 80 - BlocksRecycled: 0 - BufferedPins: 0 - BytesWritten: 0 - InactiveTotalTime: 0 - MaxBlockSize: 8388608 - MemoryLimit: 150323855360 - PeakMemoryUsage: 524288 - TotalBufferWaitTime: 0 - TotalEncryptionTime: 0 - TotalIntegrityCheckTime: 0 - TotalReadBlockTime: 0 - TotalTime: 0 CodeGen - CodegenTime: 1998098 - CompileTime: 27858199 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 208626598 - PrepareTime: 260852444 - TotalTime: 498582599 DataStreamSender (dst_id=7) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 2048 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 11423 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=6) ExecOption: Codegen Enabled - BuildTime: 7242 - GetNewBlockTime: 121676 - GetResultsTime: 0 - HTResizeTime: 3098 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 2367488 - PinTime: 0 - RowsRepartitioned: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 1465332034 - UnpinTime: 951 EXCHANGE_NODE (id=5) - BytesReceived: 0 - ConvertRowBatchTime: 1434 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 1200122929 - InactiveTotalTime: 1200124409 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 1200132179 Instance f44a3eb878bdb29e:cc59ef7f71c441bd (host=paehowchq1d02.prudential.com:22000) - AverageThreadTokens: 1.0 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 2377728 - PerHostPeakMemUsage: 144933768 - PrepareTime: 244144670 - RowsProduced: 0 - TotalCpuTime: 474591032 - TotalNetworkReceiveTime: 1229549084 - TotalNetworkSendTime: 311 - TotalStorageWaitTime: 0 - TotalTime: 1475347686 CodeGen - CodegenTime: 2055816 - CompileTime: 27585191 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 199678090 - PrepareTime: 238973031 - TotalTime: 468923303 DataStreamSender (dst_id=7) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 2048 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 12429 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=6) ExecOption: Codegen Enabled - BuildTime: 8067 - GetNewBlockTime: 118497 - GetResultsTime: 0 - HTResizeTime: 2650 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 2367488 - PinTime: 0 - RowsRepartitioned: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 1474518773 - UnpinTime: 957 EXCHANGE_NODE (id=5) - BytesReceived: 0 - ConvertRowBatchTime: 1230 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 1229544458 - InactiveTotalTime: 1229545616 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 1229553352 Fragment F00 Instance f44a3eb878bdb29e:cc59ef7f71c441bf (host=paehowchq1d03.prudential.com:22000) Hdfs split stats (:<# splits>/): 8:1/229.76 MB Filter 0 arrival: 586ms Filter 1 arrival: 586ms - AverageThreadTokens: 1.6666666666666667 - BloomFilterBytes: 2097152 - InactiveTotalTime: 0 - PeakMemoryUsage: 140422920 - PerHostPeakMemUsage: 144889608 - PrepareTime: 262474080 - RowsProduced: 0 - TotalCpuTime: 2570335981 - TotalNetworkReceiveTime: 57459548 - TotalNetworkSendTime: 1386 - TotalStorageWaitTime: 47370266 - TotalTime: 1339157060 CodeGen - CodegenTime: 1732411 - CompileTime: 44985909 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 216730791 - PrepareTime: 255883112 - TotalTime: 519660119 DataStreamSender (dst_id=5) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 6144 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 14515 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=3) ExecOption: Streaming Preaggregation, Codegen Enabled - GetNewBlockTime: 63660 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 1327104 - PinTime: 0 - ReductionFactorEstimate: 0.0 - ReductionFactorThresholdToExpand: 0.0 - RowsPassedThrough: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - StreamingTime: 1563 - TotalTime: 1336845339 - UnpinTime: 0 HASH_JOIN_NODE (id=2) ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously, Build-Side Runtime-Filter Produced - BuildPartitionTime: 3465 - BuildRows: 0 - BuildRowsPartitioned: 0 - BuildTime: 0 - GetNewBlockTime: 300447 - HashBuckets: 0 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - LocalTime: 7267552 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 10260624 - PinTime: 0 - ProbeRows: 0 - ProbeRowsPartitioned: 0 - ProbeTime: 1002 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 1076889808 - UnpinTime: 2165 EXCHANGE_NODE (id=4) - BytesReceived: 0 - ConvertRowBatchTime: 1156 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 57451535 - InactiveTotalTime: 57452705 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 57464250 HDFS_SCAN_NODE (id=0) ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 1 Hdfs split stats (:<# splits>/): 8:1/229.76 MB Runtime filters: All filters arrived. Waited 0 Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% File Formats: PARQUET/SNAPPY:5 - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 1.0 - BytesRead: 128753015 - BytesReadDataNodeCache: 0 - BytesReadLocal: 128753015 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 128753015 - DecompressionTime: 346345182 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 5 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 139089672 - PerReadThreadRawHdfsThroughput: 819529752 - RemoteScanRanges: 0 - RowsRead: 5607924 - RowsReturned: 0 - RowsReturnedRate: 0 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 1 - ScannerThreadsTotalWallClockTime: 1012881819 - MaterializeTupleTime(*): 595314139 - ScannerThreadsSysTime: 7998000 - ScannerThreadsUserTime: 956854000 - ScannerThreadsVoluntaryContextSwitches: 7 - TotalRawHdfsReadTime(*): 157105968 - TotalReadThroughput: 128753015 - TotalTime: 1012158006 Filter 0 - InactiveTotalTime: 0 - Rows processed: 5607924 - Rows rejected: 5607924 - Rows total: 5607924 - TotalTime: 0 Filter 1 - InactiveTotalTime: 0 - Rows processed: 0 - Rows rejected: 0 - Rows total: 0 - TotalTime: 0 Instance f44a3eb878bdb29e:cc59ef7f71c441be (host=paehowchq1d01.prudential.com:22000) Hdfs split stats (:<# splits>/): 3:1/217.32 MB Filter 0 arrival: 594ms Filter 1 arrival: 594ms - AverageThreadTokens: 1.6666666666666667 - BloomFilterBytes: 2097152 - InactiveTotalTime: 0 - PeakMemoryUsage: 136242272 - PerHostPeakMemUsage: 140708960 - PrepareTime: 275937720 - RowsProduced: 0 - TotalCpuTime: 2631455345 - TotalNetworkReceiveTime: 28213619 - TotalNetworkSendTime: 1186 - TotalStorageWaitTime: 37030761 - TotalTime: 1343574539 CodeGen - CodegenTime: 1825065 - CompileTime: 46191179 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 241613335 - PrepareTime: 270229073 - TotalTime: 559253997 DataStreamSender (dst_id=5) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 6144 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 13919 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=3) ExecOption: Streaming Preaggregation, Codegen Enabled - GetNewBlockTime: 34923 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 1327104 - PinTime: 0 - ReductionFactorEstimate: 0.0 - ReductionFactorThresholdToExpand: 0.0 - RowsPassedThrough: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - StreamingTime: 2296 - TotalTime: 1341681875 - UnpinTime: 0 HASH_JOIN_NODE (id=2) ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously, Build-Side Runtime-Filter Produced - BuildPartitionTime: 1254 - BuildRows: 0 - BuildRowsPartitioned: 0 - BuildTime: 0 - GetNewBlockTime: 111316 - HashBuckets: 0 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - LocalTime: 7128657 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 8433824 - PinTime: 0 - ProbeRows: 0 - ProbeRowsPartitioned: 0 - ProbeTime: 1938 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 1068110702 - UnpinTime: 1902 EXCHANGE_NODE (id=4) - BytesReceived: 0 - ConvertRowBatchTime: 744 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 28209381 - InactiveTotalTime: 28210390 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 28216702 HDFS_SCAN_NODE (id=0) ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 1 Hdfs split stats (:<# splits>/): 3:1/217.32 MB Runtime filters: All filters arrived. Waited 0 Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% File Formats: PARQUET/SNAPPY:5 - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 1.0 - BytesRead: 121863195 - BytesReadDataNodeCache: 0 - BytesReadLocal: 121863195 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 121863195 - DecompressionTime: 355224740 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 5 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 134909024 - PerReadThreadRawHdfsThroughput: 689828017 - RemoteScanRanges: 0 - RowsRead: 5188656 - RowsReturned: 0 - RowsReturnedRate: 0 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 3 - ScannerThreadsTotalWallClockTime: 1034929524 - MaterializeTupleTime(*): 619312561 - ScannerThreadsSysTime: 2999000 - ScannerThreadsUserTime: 993848000 - ScannerThreadsVoluntaryContextSwitches: 8 - TotalRawHdfsReadTime(*): 176657358 - TotalReadThroughput: 81242130 - TotalTime: 1032765343 Filter 0 - InactiveTotalTime: 0 - Rows processed: 5188656 - Rows rejected: 5188656 - Rows total: 5188656 - TotalTime: 0 Filter 1 - InactiveTotalTime: 0 - Rows processed: 0 - Rows rejected: 0 - Rows total: 0 - TotalTime: 0 Instance f44a3eb878bdb29e:cc59ef7f71c441c0 (host=paehowchq1d02.prudential.com:22000) Hdfs split stats (:<# splits>/): 3:1/242.41 MB Filter 0 arrival: 594ms Filter 1 arrival: 594ms - AverageThreadTokens: 1.75 - BloomFilterBytes: 2097152 - InactiveTotalTime: 0 - PeakMemoryUsage: 140458888 - PerHostPeakMemUsage: 144933768 - PrepareTime: 245031926 - RowsProduced: 0 - TotalCpuTime: 2757688950 - TotalNetworkReceiveTime: 84309021 - TotalNetworkSendTime: 1904 - TotalStorageWaitTime: 50629917 - TotalTime: 1438701309 CodeGen - CodegenTime: 1639519 - CompileTime: 44462405 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 217915643 - PrepareTime: 237284664 - TotalTime: 502563331 DataStreamSender (dst_id=5) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 6144 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 12465 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 AGGREGATION_NODE (id=3) ExecOption: Streaming Preaggregation, Codegen Enabled - GetNewBlockTime: 572790 - GetResultsTime: 0 - HTResizeTime: 0 - HashBuckets: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 1327104 - PinTime: 0 - ReductionFactorEstimate: 0.0 - ReductionFactorThresholdToExpand: 0.0 - RowsPassedThrough: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - StreamingTime: 2159 - TotalTime: 1436726614 - UnpinTime: 0 HASH_JOIN_NODE (id=2) ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously, Build-Side Runtime-Filter Produced - BuildPartitionTime: 2990 - BuildRows: 0 - BuildRowsPartitioned: 0 - BuildTime: 0 - GetNewBlockTime: 162467 - HashBuckets: 0 - HashCollisions: 0 - InactiveTotalTime: 0 - LargestPartitionPercent: 0 - LocalTime: 6427672 - MaxPartitionLevel: 0 - NumRepartitions: 0 - PartitionsCreated: 16 - PeakMemoryUsage: 8432768 - PinTime: 0 - ProbeRows: 0 - ProbeRowsPartitioned: 0 - ProbeTime: 1457 - RowsReturned: 0 - RowsReturnedRate: 0 - SpilledPartitions: 0 - TotalTime: 1194301717 - UnpinTime: 1780 EXCHANGE_NODE (id=4) - BytesReceived: 0 - ConvertRowBatchTime: 1167 - DeserializeRowBatchTimer: 0 - FirstBatchArrivalWaitTime: 84303511 - InactiveTotalTime: 84304685 - PeakMemoryUsage: 0 - RowsReturned: 0 - RowsReturnedRate: 0 - SendersBlockedTimer: 0 - SendersBlockedTotalTimer(*): 0 - TotalTime: 84314455 HDFS_SCAN_NODE (id=0) ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 1 Hdfs split stats (:<# splits>/): 3:1/242.41 MB Runtime filters: All filters arrived. Waited 0 Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% File Formats: PARQUET/SNAPPY:5 - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.6666666666666666 - BytesRead: 135593992 - BytesReadDataNodeCache: 0 - BytesReadLocal: 135593992 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 135593992 - DecompressionTime: 373631268 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 5 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 139073704 - PerReadThreadRawHdfsThroughput: 709238153 - RemoteScanRanges: 0 - RowsRead: 5973813 - RowsReturned: 0 - RowsReturnedRate: 0 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 3 - ScannerThreadsTotalWallClockTime: 1104479876 - MaterializeTupleTime(*): 654384481 - ScannerThreadsSysTime: 9998000 - ScannerThreadsUserTime: 1040841000 - ScannerThreadsVoluntaryContextSwitches: 7 - TotalRawHdfsReadTime(*): 191182597 - TotalReadThroughput: 90395994 - TotalTime: 1103559590 Filter 0 - InactiveTotalTime: 0 - Rows processed: 5973813 - Rows rejected: 5973813 - Rows total: 5973813 - TotalTime: 0 Filter 1 - InactiveTotalTime: 0 - Rows processed: 0 - Rows rejected: 0 - Rows total: 0 - TotalTime: 0 Fragment F01 Instance f44a3eb878bdb29e:cc59ef7f71c441c1 (host=paehowchq1d01.prudential.com:22000) Hdfs split stats (:<# splits>/): 9:1/200.05 MB - AverageThreadTokens: 1.0 - BloomFilterBytes: 0 - InactiveTotalTime: 0 - PeakMemoryUsage: 26516552 - PerHostPeakMemUsage: 30219336 - PrepareTime: 262373045 - RowsProduced: 0 - TotalCpuTime: 722171551 - TotalNetworkReceiveTime: 0 - TotalNetworkSendTime: 1623 - TotalStorageWaitTime: 16915604 - TotalTime: 409819215 CodeGen - CodegenTime: 0 - CompileTime: 1471491 - InactiveTotalTime: 0 - LoadTime: 0 - ModuleBitcodeSize: 1925128 - OptimizationTime: 181009760 - PrepareTime: 260804255 - TotalTime: 444347099 DataStreamSender (dst_id=4) - BytesSent: 0 - InactiveTotalTime: 0 - NetworkThroughput(*): 0 - OverallThroughput: 0 - PeakMemoryUsage: 8184 - RowsReturned: 0 - SerializeBatchTime: 0 - TotalTime: 24995 - TransmitDataRPCTime: 0 - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=1) ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 1 Hdfs split stats (:<# splits>/): 9:1/200.05 MB Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% File Formats: PARQUET/SNAPPY:3 - AverageHdfsReadThreadConcurrency: 0.0 - AverageScannerThreadConcurrency: 0.0 - BytesRead: 13557142 - BytesReadDataNodeCache: 0 - BytesReadLocal: 13557142 - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 13557142 - DecompressionTime: 23086127 - InactiveTotalTime: 0 - MaxCompressedTextFileLength: 0 - NumColumns: 3 - NumDisksAccessed: 1 - NumRowGroups: 1 - NumScannerThreadsStarted: 1 - PeakMemoryUsage: 26442824 - PerReadThreadRawHdfsThroughput: 1156141021 - RemoteScanRanges: 0 - RowsRead: 1463085 - RowsReturned: 0 - RowsReturnedRate: 0 - ScanRangesComplete: 1 - ScannerThreadsInvoluntaryContextSwitches: 0 - ScannerThreadsTotalWallClockTime: 145863138 - MaterializeTupleTime(*): 103748688 - ScannerThreadsSysTime: 3999000 - ScannerThreadsUserTime: 123981000 - ScannerThreadsVoluntaryContextSwitches: 8 - TotalRawHdfsReadTime(*): 11726201 - TotalReadThroughput: 0 - TotalTime: 670389734