Query (id=9793b46973e34c54:b203a98b8356ae9e): Summary: Start Time: 2013-04-29 12:11:31 End Time: 2013-04-29 12:15:54 Query Type: DML Query State: EXCEPTION Impala Version: impalad version 1.0 RELEASE (build 3b6063a882fa820a78082c52d62862000ad84cc1) Built on Fri, 26 Apr 2013 10:58:30 PST User: root Default Db: default Sql Statement: insert OVERWRITE logs_par PARTITION (customeruuid,year,month) SELECT customerkey,clustername,service,roletype,role,host,filename,ts,dt,level,class,message,customeruuid,year,month FROM logs_txt Plan: ---------------- PLAN FRAGMENT 0 PARTITION: HASH_PARTITIONED: customeruuid, year, month WRITE TO HDFS table=default.logs_par overwrite=true partitions: customeruuid,year,month 1:EXCHANGE tuple ids: 0 PLAN FRAGMENT 1 PARTITION: RANDOM STREAM DATA SINK EXCHANGE ID: 1 HASH_PARTITIONED: customeruuid, year, month 0:SCAN HDFS table=default.logs_txt #partitions=56 size=14.67GB tuple ids: 0 ---------------- Query Timeline: 4m22s - Start execution: 3.295us (3.295us) - Planning finished: 393.641ms (393.638ms) - Rows available: 4m22s (4m21s) - Unregister query: 4m22s (31.378ms) ImpalaServer: - RowMaterializationTimer: 0ns Execution Profile 9793b46973e34c54:b203a98b8356ae9e:(Active: 4m21s, % non-child: 0.00%) - FinalizationTimer: 0ns Fragment 0: Instance 9793b46973e34c54:b203a98b8356aea3 (host=forza-3.cloud.rtp.cloudera.com:22000):(Active: 4m10s, % non-child: 0.00%) - AverageThreadTokens: 1.00 - RowsProduced: 10.62M (10615808) CodeGen:(Active: 82.121ms, % non-child: 0.03%) - CodegenTime: 0ns - CompileTime: 73.746ms - LoadTime: 8.372ms - ModuleFileSize: 65.23 KB HdfsTableSink:(Active: 1m20s, % non-child: 32.18%) - EncodeTimer: 36s791ms - HdfsWriteTimer: 12s719ms - MemoryUsed: 1.17 GB - RowsInserted: 6.73M (6725630) EXCHANGE_NODE (id=1):(Active: 2m49s, % non-child: 67.80%) - BytesReceived: 1.48 GB - ConvertRowBatchTime: 495.116ms - DataArrivalWaitTime: 2m53s - DeserializeRowBatchTimer: 15s031ms - FirstBatchArrivalWaitTime: 0ns - MemoryUsed: 0.00 - RowsReturned: 10.62M (10616755) - RowsReturnedRate: 62.57 K/sec - SendersBlockedTimer: 13s139ms - SendersBlockedTotalTimer(*): 50s052ms Instance 9793b46973e34c54:b203a98b8356aea0 (host=forza-2.cloud.rtp.cloudera.com:22000):(Active: 3m3s, % non-child: 0.00%) - AverageThreadTokens: 1.00 - RowsProduced: 1.29M (1289216) CodeGen:(Active: 82.501ms, % non-child: 0.05%) - CodegenTime: 0ns - CompileTime: 74.731ms - LoadTime: 7.765ms - ModuleFileSize: 65.23 KB HdfsTableSink:(Active: 15s323ms, % non-child: 8.37%) - EncodeTimer: 11s283ms - HdfsWriteTimer: 0ns - MemoryUsed: 0.00 - RowsInserted: 0 EXCHANGE_NODE (id=1):(Active: 2m47s, % non-child: 91.63%) - BytesReceived: 211.29 MB - ConvertRowBatchTime: 68.721ms - DataArrivalWaitTime: 2m47s - DeserializeRowBatchTimer: 2s181ms - FirstBatchArrivalWaitTime: 0ns - MemoryUsed: 0.00 - RowsReturned: 1.29M (1289477) - RowsReturnedRate: 7.69 K/sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 9793b46973e34c54:b203a98b8356ae9f (host=forza-5.cloud.rtp.cloudera.com:22000):(Active: 2m31s, % non-child: 0.00%) - AverageThreadTokens: 1.00 - RowsProduced: 12.21M (12211200) CodeGen:(Active: 84.49ms, % non-child: 0.06%) - CodegenTime: 0ns - CompileTime: 73.561ms - LoadTime: 10.484ms - ModuleFileSize: 65.23 KB HdfsTableSink:(Active: 1m34s, % non-child: 62.51%) - EncodeTimer: 41s914ms - HdfsWriteTimer: 15s815ms - MemoryUsed: 1.17 GB - RowsInserted: 8.68M (8681968) EXCHANGE_NODE (id=1):(Active: 56s581ms, % non-child: 37.45%) - BytesReceived: 1.78 GB - ConvertRowBatchTime: 617.31ms - DataArrivalWaitTime: 1m25s - DeserializeRowBatchTimer: 16s883ms - FirstBatchArrivalWaitTime: 0ns - MemoryUsed: 0.00 - RowsReturned: 12.21M (12211794) - RowsReturnedRate: 215.83 K/sec - SendersBlockedTimer: 23s537ms - SendersBlockedTotalTimer(*): 1m37s Instance 9793b46973e34c54:b203a98b8356aea1 (host=forza-6.cloud.rtp.cloudera.com:22000):(Active: 2m28s, % non-child: 0.00%) - AverageThreadTokens: 1.00 - RowsProduced: 598.02K (598016) CodeGen:(Active: 100.943ms, % non-child: 0.07%) - CodegenTime: 0ns - CompileTime: 78.990ms - LoadTime: 21.948ms - ModuleFileSize: 65.23 KB HdfsTableSink:(Active: 3s430ms, % non-child: 2.32%) - EncodeTimer: 1s611ms - HdfsWriteTimer: 0ns - MemoryUsed: 0.00 - RowsInserted: 0 EXCHANGE_NODE (id=1):(Active: 2m24s, % non-child: 97.68%) - BytesReceived: 69.51 MB - ConvertRowBatchTime: 24.949ms - DataArrivalWaitTime: 2m59s - DeserializeRowBatchTimer: 639.535ms - FirstBatchArrivalWaitTime: 0ns - MemoryUsed: 0.00 - RowsReturned: 598.73K (598734) - RowsReturnedRate: 4.14 K/sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Instance 9793b46973e34c54:b203a98b8356aea2 (host=forza-4.cloud.rtp.cloudera.com:22000):(Active: 2m9s, % non-child: 0.00%) - AverageThreadTokens: 1.00 - RowsProduced: 1.94M (1938432) CodeGen:(Active: 86.662ms, % non-child: 0.07%) - CodegenTime: 0ns - CompileTime: 78.171ms - LoadTime: 8.486ms - ModuleFileSize: 65.23 KB HdfsTableSink:(Active: 15s408ms, % non-child: 11.89%) - EncodeTimer: 9s278ms - HdfsWriteTimer: 0ns - MemoryUsed: 0.00 - RowsInserted: 0 EXCHANGE_NODE (id=1):(Active: 1m54s, % non-child: 88.10%) - BytesReceived: 233.37 MB - ConvertRowBatchTime: 103.385ms - DataArrivalWaitTime: 1m53s - DeserializeRowBatchTimer: 3s182ms - FirstBatchArrivalWaitTime: 0ns - MemoryUsed: 0.00 - RowsReturned: 1.94M (1938432) - RowsReturnedRate: 16.98 K/sec - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Fragment 1: Instance 9793b46973e34c54:b203a98b8356aea7 (host=forza-4.cloud.rtp.cloudera.com:22000):(Active: 4m10s, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:29/2.93 GB - AverageThreadTokens: 5.89 - RowsProduced: 5.00M (5002392) CodeGen:(Active: 127.355ms, % non-child: 0.05%) - CodegenTime: 5.215ms - CompileTime: 117.605ms - LoadTime: 9.745ms - ModuleFileSize: 65.23 KB DataStreamSender (dst_id=1):(Active: 4m7s, % non-child: 99.13%) - BytesSent: 716.43 MB - NetworkThroughput(*): 4.67 MB/sec - OverallThroughput: 2.89 MB/sec - SerializeBatchTime: 16s931ms - ThriftTransmitTime(*): 2m33s - UncompressedRowBatchSize: 3.75 GB HDFS_SCAN_NODE (id=0):(Active: 2s138ms, % non-child: 0.85%) Hdfs split stats (:<# splits>/): 0:29/2.93 GB - AverageHdfsReadThreadConcurrency: 0.28 - HdfsReadThreadConcurrencyCountPercentage=0: 0.00 - HdfsReadThreadConcurrencyCountPercentage=1: 0.00 - HdfsReadThreadConcurrencyCountPercentage=2: 0.00 - AverageIoMgrQueueCapcity: 256.00 - AverageIoMgrQueueSize: 8.45 - AverageScannerThreadConcurrency: 4.51 - BytesRead: 2.67 GB - MemoryUsed: 294.73 KB - NumDisksAccessed: 0 - PerReadThreadRawHdfsThroughput: 40.82 MB/sec - RowsRead: 5.02M (5022872) - RowsReturned: 5.00M (5002392) - RowsReturnedRate: 2.34 M/sec - ScanRangesComplete: 25 - ScannerThreadsInvoluntaryContextSwitches: 3.12K (3124) - ScannerThreadsTotalWallClockTime: 16m35s - DelimiterParseTime: 1m28s - MaterializeTupleTime(*): 1s485ms - ScannerThreadsSysTime: 5s224ms - ScannerThreadsUserTime: 32s230ms - ScannerThreadsVoluntaryContextSwitches: 7.02K (7021) - TotalRawHdfsReadTime(*): 1m6s - TotalReadThroughput: 10.74 MB/sec Instance 9793b46973e34c54:b203a98b8356aea5 (host=forza-2.cloud.rtp.cloudera.com:22000):(Active: 3m2s, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:32/2.92 GB - AverageThreadTokens: 4.48 - RowsProduced: 5.75M (5745655) CodeGen:(Active: 125.552ms, % non-child: 0.07%) - CodegenTime: 5.604ms - CompileTime: 112.828ms - LoadTime: 12.717ms - ModuleFileSize: 65.23 KB DataStreamSender (dst_id=1):(Active: 2m58s, % non-child: 97.36%) - BytesSent: 814.27 MB - NetworkThroughput(*): 5.47 MB/sec - OverallThroughput: 4.57 MB/sec - SerializeBatchTime: 18s486ms - ThriftTransmitTime(*): 2m28s - UncompressedRowBatchSize: 4.26 GB HDFS_SCAN_NODE (id=0):(Active: 4s780ms, % non-child: 2.61%) Hdfs split stats (:<# splits>/): 0:32/2.92 GB File Formats: TEXT/NONE:32 ExecOption: Codegen enabled: 32 out of 32 - AverageHdfsReadThreadConcurrency: 0.04 - HdfsReadThreadConcurrencyCountPercentage=0: 96.45 - HdfsReadThreadConcurrencyCountPercentage=1: 3.55 - HdfsReadThreadConcurrencyCountPercentage=2: 0.00 - AverageIoMgrQueueCapcity: 256.00 - AverageIoMgrQueueSize: 13.88 - AverageScannerThreadConcurrency: 4.70 - BytesRead: 2.92 GB - MemoryUsed: 368.63 KB - NumDisksAccessed: 1 - PerReadThreadRawHdfsThroughput: 467.49 MB/sec - RowsRead: 5.75M (5745655) - RowsReturned: 5.75M (5745655) - RowsReturnedRate: 1.20 M/sec - ScanRangesComplete: 32 - ScannerThreadsInvoluntaryContextSwitches: 3.38K (3385) - ScannerThreadsTotalWallClockTime: 14m29s - DelimiterParseTime: 41s417ms - MaterializeTupleTime(*): 754.961ms - ScannerThreadsSysTime: 4s599ms - ScannerThreadsUserTime: 37s715ms - ScannerThreadsVoluntaryContextSwitches: 7.16K (7164) - TotalRawHdfsReadTime(*): 6s396ms - TotalReadThroughput: 16.34 MB/sec Instance 9793b46973e34c54:b203a98b8356aea6 (host=forza-6.cloud.rtp.cloudera.com:22000):(Active: 2m50s, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:35/2.97 GB - AverageThreadTokens: 4.18 - RowsProduced: 5.54M (5538860) CodeGen:(Active: 114.294ms, % non-child: 0.07%) - CodegenTime: 6.173ms - CompileTime: 105.688ms - LoadTime: 8.600ms - ModuleFileSize: 65.23 KB DataStreamSender (dst_id=1):(Active: 2m48s, % non-child: 98.57%) - BytesSent: 801.02 MB - NetworkThroughput(*): 5.66 MB/sec - OverallThroughput: 4.76 MB/sec - SerializeBatchTime: 18s167ms - ThriftTransmitTime(*): 2m21s - UncompressedRowBatchSize: 4.26 GB HDFS_SCAN_NODE (id=0):(Active: 2s415ms, % non-child: 1.41%) Hdfs split stats (:<# splits>/): 0:35/2.97 GB File Formats: TEXT/NONE:35 ExecOption: Codegen enabled: 35 out of 35 - AverageHdfsReadThreadConcurrency: 0.04 - HdfsReadThreadConcurrencyCountPercentage=0: 96.19 - HdfsReadThreadConcurrencyCountPercentage=1: 3.81 - AverageIoMgrQueueCapcity: 256.00 - AverageIoMgrQueueSize: 12.07 - AverageScannerThreadConcurrency: 4.51 - BytesRead: 2.97 GB - MemoryUsed: 528.88 KB - NumDisksAccessed: 1 - PerReadThreadRawHdfsThroughput: 538.14 MB/sec - RowsRead: 5.54M (5538860) - RowsReturned: 5.54M (5538860) - RowsReturnedRate: 2.29 M/sec - ScanRangesComplete: 35 - ScannerThreadsInvoluntaryContextSwitches: 2.77K (2765) - ScannerThreadsTotalWallClockTime: 12m51s - DelimiterParseTime: 42s517ms - MaterializeTupleTime(*): 641.351ms - ScannerThreadsSysTime: 3s843ms - ScannerThreadsUserTime: 39s525ms - ScannerThreadsVoluntaryContextSwitches: 7.20K (7201) - TotalRawHdfsReadTime(*): 5s645ms - TotalReadThroughput: 17.77 MB/sec Instance 9793b46973e34c54:b203a98b8356aea4 (host=forza-5.cloud.rtp.cloudera.com:22000):(Active: 2m13s, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:30/2.90 GB - AverageThreadTokens: 5.51 - RowsProduced: 5.16M (5163306) CodeGen:(Active: 190.94ms, % non-child: 0.14%) - CodegenTime: 9.23ms - CompileTime: 176.959ms - LoadTime: 13.131ms - ModuleFileSize: 65.23 KB DataStreamSender (dst_id=1):(Active: 2m11s, % non-child: 98.45%) - BytesSent: 754.98 MB - NetworkThroughput(*): 7.90 MB/sec - OverallThroughput: 5.75 MB/sec - SerializeBatchTime: 18s106ms - ThriftTransmitTime(*): 1m35s - UncompressedRowBatchSize: 3.98 GB HDFS_SCAN_NODE (id=0):(Active: 2s032ms, % non-child: 1.52%) Hdfs split stats (:<# splits>/): 0:30/2.90 GB - AverageHdfsReadThreadConcurrency: 0.04 - HdfsReadThreadConcurrencyCountPercentage=0: 0.00 - HdfsReadThreadConcurrencyCountPercentage=1: 0.00 - AverageIoMgrQueueCapcity: 256.00 - AverageIoMgrQueueSize: 7.45 - AverageScannerThreadConcurrency: 4.38 - BytesRead: 2.90 GB - MemoryUsed: 343.89 KB - NumDisksAccessed: 0 - PerReadThreadRawHdfsThroughput: 283.04 MB/sec - RowsRead: 5.17M (5173546) - RowsReturned: 5.16M (5163306) - RowsReturnedRate: 2.54 M/sec - ScanRangesComplete: 26 - ScannerThreadsInvoluntaryContextSwitches: 8.00K (8004) - ScannerThreadsTotalWallClockTime: 9m17s - DelimiterParseTime: 40s985ms - MaterializeTupleTime(*): 805.786ms - ScannerThreadsSysTime: 6s444ms - ScannerThreadsUserTime: 29s578ms - ScannerThreadsVoluntaryContextSwitches: 5.55K (5551) - TotalRawHdfsReadTime(*): 10s486ms - TotalReadThroughput: 11.53 MB/sec Instance 9793b46973e34c54:b203a98b8356aea8 (host=forza-3.cloud.rtp.cloudera.com:22000):(Active: 2m13s, % non-child: 0.00%) Hdfs split stats (:<# splits>/): 0:32/2.96 GB - AverageThreadTokens: 5.46 - RowsProduced: 5.29M (5285834) CodeGen:(Active: 180.385ms, % non-child: 0.14%) - CodegenTime: 9.458ms - CompileTime: 167.361ms - LoadTime: 13.18ms - ModuleFileSize: 65.23 KB DataStreamSender (dst_id=1):(Active: 2m10s, % non-child: 98.13%) - BytesSent: 762.27 MB - NetworkThroughput(*): 7.68 MB/sec - OverallThroughput: 5.82 MB/sec - SerializeBatchTime: 17s829ms - ThriftTransmitTime(*): 1m39s - UncompressedRowBatchSize: 4.05 GB HDFS_SCAN_NODE (id=0):(Active: 2s451ms, % non-child: 1.84%) Hdfs split stats (:<# splits>/): 0:32/2.96 GB - AverageHdfsReadThreadConcurrency: 0.02 - HdfsReadThreadConcurrencyCountPercentage=0: 0.00 - HdfsReadThreadConcurrencyCountPercentage=1: 0.00 - HdfsReadThreadConcurrencyCountPercentage=2: 0.00 - AverageIoMgrQueueCapcity: 256.00 - AverageIoMgrQueueSize: 6.70 - AverageScannerThreadConcurrency: 4.40 - BytesRead: 2.96 GB - MemoryUsed: 354.31 KB - NumDisksAccessed: 0 - PerReadThreadRawHdfsThroughput: 495.13 MB/sec - RowsRead: 5.31M (5307137) - RowsReturned: 5.29M (5285834) - RowsReturnedRate: 2.16 M/sec - ScanRangesComplete: 28 - ScannerThreadsInvoluntaryContextSwitches: 4.22K (4223) - ScannerThreadsTotalWallClockTime: 9m34s - DelimiterParseTime: 42s182ms - MaterializeTupleTime(*): 731.381ms - ScannerThreadsSysTime: 4s967ms - ScannerThreadsUserTime: 32s348ms - ScannerThreadsVoluntaryContextSwitches: 6.00K (5999) - TotalRawHdfsReadTime(*): 6s114ms - TotalReadThroughput: 11.67 MB/sec