Query (id=6b417e40e3a54863:84ca6e5163f057ab):
  Summary:
    Session ID: 2e4fca32251675cc:8c138d054c2c6387
    Session Type: HIVESERVER2
    HiveServer2 Protocol Version: V1
    Start Time: 2015-02-13 12:14:24.172597000
    End Time: 2015-02-13 12:20:02.819836000
    Query Type: DML
    Query State: EXCEPTION
    Query Status: Memory limit exceeded
    Impala Version: impalad version 2.1.1-cdh5 RELEASE (build 7901877736e29716147c4804b0841afc4ebc9037)
    User: 
    Connected User: 
    Delegated User: 
    Network Address: ::ffff:192.168.103.98:49969
    Default Db: default
    Sql Statement: insert into table DWH.FactDailyUser
partition(dt)
select *, `Date` from default.factdailyuser3
    Coordinator: node02:22000
    Plan: 
----------------
Estimated Per-Host Requirements: Memory=2.81GB VCores=1
WARNING: The following tables are missing relevant table and/or column statistics.
default.factdailyuser3

F00:PLAN FRAGMENT [RANDOM]
  WRITE TO HDFS [dwh.factdailyuser, OVERWRITE=false, PARTITION-KEYS=(`Date`)]
  |  partitions=unavailable
  |  hosts=1 per-host-mem=518B
  |
  00:SCAN HDFS [default.factdailyuser3, RANDOM]
     partitions=1/1 files=10 size=1.00TB
     table stats: 0 rows total
     column stats: unavailable
     hosts=1 per-host-mem=2.81GB
     tuple-ids=0 row-size=518B cardinality=0
----------------
    Estimated Per-Host Mem: 3019899398
    Estimated Per-Host VCores: 1
    Tables Missing Stats: default.factdailyuser3
    Request Pool: default-pool
    ExecSummary: 
Operator       #Hosts   Avg Time  Max Time   #Rows  Est. #Rows  Peak Mem  Est. Peak Mem  Detail                 
----------------------------------------------------------------------------------------------------------------
00:SCAN HDFS        3  523.651ms  570.24ms  75.81M           0   1.14 GB        2.81 GB  default.factdailyuser3 
    Query Timeline: 5m38s
       - Start execution: 87.779us (87.779us)
       - Planning finished: 59.832ms (59.744ms)
       - Ready to start remote fragments: 71.578ms (11.746ms)
       - Remote fragments started: 287.77ms (215.499ms)
       - Request finished: 5m38s (5m38s)
       - Unregister query: 5m38s (1.236ms)
  ImpalaServer:
     - ClientFetchWaitTimer: 0ns
     - RowMaterializationTimer: 0ns
  Execution Profile 6b417e40e3a54863:84ca6e5163f057ab:(Total: 5m38s, non-child: 0ns, % non-child: 0.00%)
    Fragment start latencies:  count: 3, last: 0.209172, min: 0.206827, max: 0.209172, mean: 0.208096, stddev: 0.000967148
    Per Node Peak Memory Usage: node01:22000(18.14 GB) node02:22000(5.51 GB) node03:22000(5.40 GB) 
     - FinalizationTimer: 78.293ms
    Averaged Fragment F00:(Total: 5m34s, non-child: 0ns, % non-child: 0.00%)
      split sizes:  min: 342.31 GB, max: 342.38 GB, avg: 342.35 GB, stddev: 29.98 MB
      completion times: min:5m38s  max:5m38s  mean: 5m38s  stddev:1.454ms
      execution rates: min:1.01 GB/sec  max:1.01 GB/sec  mean:1.01 GB/sec  stddev:93.95 KB/sec
      num instances: 3
       - AverageThreadTokens: 35.77 
       - PeakMemoryUsage: 9.68 GB (10396792149)
       - PerHostPeakMemUsage: 9.68 GB (10396792149)
       - PrepareTime: 194.493ms
       - RowsProduced: 25.27M (25268540)
       - TotalCpuTime: 0ns
       - TotalNetworkReceiveTime: 0ns
       - TotalNetworkSendTime: 0ns
       - TotalStorageWaitTime: 1m6s
      BlockMgr:
         - BlockWritesOutstanding: 0
         - BlocksCreated: 0
         - BlocksRecycled: 0
         - BufferedPins: 0
         - BytesWritten: 0
         - MaxBlockSize: 8.00 MB (8388608)
         - MemoryLimit: 38.39 GB (41225814698)
         - PeakMemoryUsage: 0
         - TotalBufferWaitTime: 0ns
         - TotalEncryptionTime: 0ns
         - TotalIntegrityCheckTime: 0ns
         - TotalReadBlockTime: 0ns
      CodeGen:(Total: 2s095ms, non-child: 2s095ms, % non-child: 100.00%)
         - CodegenTime: 24.710ms
         - CompileTime: 673.887ms
         - LoadTime: 27.679us
         - ModuleFileSize: 1.98 MB (2073272)
         - OptimizationTime: 1s249ms
         - PrepareTime: 170.163ms
      HdfsTableSink:(Total: 5m34s, non-child: 5m34s, % non-child: 100.00%)
         - BytesWritten: 1.54 KB (1573)
         - CompressTimer: 12s024ms
         - EncodeTimer: 4m54s
         - FilesCreated: 393
         - HdfsWriteTimer: 0ns
         - PartitionsCreated: 393
         - PeakMemoryUsage: 8.76 GB (9407619072)
         - RowsInserted: 0
         - TmpFileCreateTimer: 3s822ms
      HDFS_SCAN_NODE (id=0):(Total: 523.651ms, non-child: 523.651ms, % non-child: 100.00%)
         - AverageHdfsReadThreadConcurrency: 0.26 
         - AverageScannerThreadConcurrency: 35.00 
         - BytesRead: 12.65 GB (13581228032)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 0
         - BytesReadRemoteUnexpected: 0
         - BytesReadShortCircuit: 0
         - DecompressionTime: 0ns
         - MaxCompressedTextFileLength: 0
         - NumDisksAccessed: 0
         - NumScannerThreadsStarted: 35
         - PeakMemoryUsage: 1.13 GB (1216495616)
         - PerReadThreadRawHdfsThroughput: 157.35 MB/sec
         - RemoteScanRanges: 0
         - RowsRead: 25.33M (25334759)
         - RowsReturned: 25.27M (25268540)
         - RowsReturnedRate: 48.54 M/sec
         - ScanRangesComplete: 70
         - ScannerThreadsInvoluntaryContextSwitches: 0
         - ScannerThreadsTotalWallClockTime: 0ns
           - DelimiterParseTime: 37s444ms
           - MaterializeTupleTime(*): 1m21s
           - ScannerThreadsSysTime: 0ns
           - ScannerThreadsUserTime: 0ns
         - ScannerThreadsVoluntaryContextSwitches: 0
         - TotalRawHdfsReadTime(*): 1m25s
         - TotalReadThroughput: 38.43 MB/sec
    Fragment F00:
      Instance 6b417e40e3a54863:84ca6e5163f057ad (host=node02:22000):(Total: 5m36s, non-child: 0ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2739/342.38 GB 
        MemoryUsage(8s000ms): 893.72 MB, 2.85 GB, 2.70 GB, 2.74 GB, 2.78 GB, 2.83 GB, 2.89 GB, 2.94 GB, 3.02 GB, 3.05 GB, 3.09 GB, 3.13 GB, 3.25 GB, 3.35 GB, 3.43 GB, 3.48 GB, 3.35 GB, 3.14 GB, 3.18 GB, 3.75 GB, 3.71 GB, 3.76 GB, 3.82 GB, 3.89 GB, 3.97 GB, 4.02 GB, 4.12 GB, 4.19 GB, 4.25 GB, 4.32 GB, 4.33 GB, 4.11 GB, 4.23 GB, 4.70 GB, 4.69 GB, 4.84 GB, 4.91 GB, 4.97 GB, 5.11 GB, 5.27 GB, 5.40 GB, 5.43 GB
        ThreadUsage(8s000ms): 25, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36
         - AverageThreadTokens: 35.74 
         - PeakMemoryUsage: 5.51 GB (5917220864)
         - PerHostPeakMemUsage: 5.51 GB (5917220864)
         - PrepareTime: 194.913ms
         - RowsProduced: 25.52M (25524757)
         - TotalCpuTime: 0ns
         - TotalNetworkReceiveTime: 0ns
         - TotalNetworkSendTime: 0ns
         - TotalStorageWaitTime: 1m15s
        BlockMgr:
           - BlockWritesOutstanding: 0
           - BlocksCreated: 0
           - BlocksRecycled: 0
           - BufferedPins: 0
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 43.20 GB (46385647616)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0ns
           - TotalEncryptionTime: 0ns
           - TotalIntegrityCheckTime: 0ns
           - TotalReadBlockTime: 0ns
        CodeGen:(Total: 2s088ms, non-child: 2s088ms, % non-child: 100.00%)
           - CodegenTime: 24.23ms
           - CompileTime: 672.838ms
           - LoadTime: 35.626us
           - ModuleFileSize: 1.98 MB (2073272)
           - OptimizationTime: 1s243ms
           - PrepareTime: 170.938ms
        HdfsTableSink:(Total: 5m35s, non-child: 5m35s, % non-child: 100.00%)
           - BytesWritten: 456.00 B (456)
           - CompressTimer: 12s625ms
           - EncodeTimer: 5m
           - FilesCreated: 114
           - HdfsWriteTimer: 0ns
           - PartitionsCreated: 114
           - PeakMemoryUsage: 4.59 GB (4924563456)
           - RowsInserted: 0
           - TmpFileCreateTimer: 1s158ms
        HDFS_SCAN_NODE (id=0):(Total: 570.24ms, non-child: 570.24ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2739/342.38 GB 
          BytesRead(8s000ms): 314.00 MB, 1.09 GB, 1.16 GB, 1.39 GB, 1.60 GB, 1.84 GB, 2.07 GB, 2.32 GB, 2.56 GB, 2.81 GB, 3.06 GB, 3.31 GB, 3.56 GB, 3.80 GB, 4.04 GB, 4.27 GB, 4.38 GB, 4.39 GB, 4.61 GB, 5.41 GB, 5.58 GB, 5.88 GB, 6.19 GB, 6.49 GB, 6.81 GB, 7.13 GB, 7.45 GB, 7.78 GB, 8.11 GB, 8.44 GB, 8.72 GB, 8.77 GB, 9.17 GB, 9.88 GB, 10.12 GB, 10.46 GB, 10.80 GB, 11.13 GB, 11.47 GB, 11.81 GB, 12.16 GB, 12.51 GB
           - AverageHdfsReadThreadConcurrency: 0.31 
           - AverageScannerThreadConcurrency: 35.00 
           - BytesRead: 12.77 GB (13715445760)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 0ns
           - MaxCompressedTextFileLength: 0
           - NumDisksAccessed: 0
           - NumScannerThreadsStarted: 35
           - PeakMemoryUsage: 1.14 GB (1224654848)
           - PerReadThreadRawHdfsThroughput: 128.53 MB/sec
           - RemoteScanRanges: 0
           - RowsRead: 25.59M (25591317)
           - RowsReturned: 25.52M (25524757)
           - RowsReturnedRate: 44.78 M/sec
           - ScanRangesComplete: 70
           - ScannerThreadsInvoluntaryContextSwitches: 0
           - ScannerThreadsTotalWallClockTime: 0ns
             - DelimiterParseTime: 37s333ms
             - MaterializeTupleTime(*): 1m20s
             - ScannerThreadsSysTime: 0ns
             - ScannerThreadsUserTime: 0ns
           - ScannerThreadsVoluntaryContextSwitches: 0
           - TotalRawHdfsReadTime(*): 1m41s
           - TotalReadThroughput: 38.64 MB/sec
      Instance 6b417e40e3a54863:84ca6e5163f057ae (host=node01:22000):(Total: 5m35s, non-child: 0ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2740/342.37 GB 
        MemoryUsage(8s000ms): 1.12 GB, 2.83 GB, 2.69 GB, 2.72 GB, 2.76 GB, 2.82 GB, 2.87 GB, 2.93 GB, 2.98 GB, 3.05 GB, 3.14 GB, 3.22 GB, 3.28 GB, 3.32 GB, 3.32 GB, 3.14 GB, 3.12 GB, 3.67 GB, 3.67 GB, 3.72 GB, 3.78 GB, 3.87 GB, 3.97 GB, 4.06 GB, 4.12 GB, 4.16 GB, 4.21 GB, 4.21 GB, 4.15 GB, 3.94 GB, 4.83 GB, 10.38 GB, 15.64 GB, 17.18 GB, 17.44 GB, 17.60 GB, 17.67 GB, 17.74 GB, 17.81 GB, 17.92 GB, 17.98 GB, 18.05 GB
        ThreadUsage(8s000ms): 27, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36
         - AverageThreadTokens: 35.79 
         - PeakMemoryUsage: 18.14 GB (19477520384)
         - PerHostPeakMemUsage: 18.14 GB (19477520384)
         - PrepareTime: 194.431ms
         - RowsProduced: 24.14M (24136595)
         - TotalCpuTime: 0ns
         - TotalNetworkReceiveTime: 0ns
         - TotalNetworkSendTime: 0ns
         - TotalStorageWaitTime: 1m1s
        BlockMgr:
           - BlockWritesOutstanding: 0
           - BlocksCreated: 0
           - BlocksRecycled: 0
           - BufferedPins: 0
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 28.78 GB (30906148864)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0ns
           - TotalEncryptionTime: 0ns
           - TotalIntegrityCheckTime: 0ns
           - TotalReadBlockTime: 0ns
        CodeGen:(Total: 2s080ms, non-child: 2s080ms, % non-child: 100.00%)
           - CodegenTime: 25.97ms
           - CompileTime: 667.667ms
           - LoadTime: 24.331us
           - ModuleFileSize: 1.98 MB (2073272)
           - OptimizationTime: 1s241ms
           - PrepareTime: 169.818ms
        HdfsTableSink:(Total: 5m34s, non-child: 5m34s, % non-child: 100.00%)
           - BytesWritten: 3.72 KB (3808)
           - CompressTimer: 10s479ms
           - EncodeTimer: 4m46s
           - FilesCreated: 952
           - HdfsWriteTimer: 0ns
           - PartitionsCreated: 952
           - PeakMemoryUsage: 17.22 GB (18488025088)
           - RowsInserted: 0
           - TmpFileCreateTimer: 9s026ms
        HDFS_SCAN_NODE (id=0):(Total: 458.182ms, non-child: 458.182ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2740/342.37 GB 
          BytesRead(8s000ms): 359.00 MB, 1.10 GB, 1.21 GB, 1.46 GB, 1.70 GB, 1.96 GB, 2.22 GB, 2.48 GB, 2.75 GB, 3.02 GB, 3.30 GB, 3.58 GB, 3.87 GB, 4.14 GB, 4.37 GB, 4.39 GB, 4.57 GB, 5.37 GB, 5.61 GB, 5.92 GB, 6.23 GB, 6.55 GB, 6.87 GB, 7.20 GB, 7.52 GB, 7.84 GB, 8.17 GB, 8.49 GB, 8.74 GB, 8.80 GB, 9.21 GB, 9.70 GB, 9.84 GB, 9.88 GB, 10.07 GB, 10.33 GB, 10.58 GB, 10.84 GB, 11.11 GB, 11.38 GB, 11.66 GB, 11.94 GB
           - AverageHdfsReadThreadConcurrency: 0.18 
           - AverageScannerThreadConcurrency: 35.00 
           - BytesRead: 12.13 GB (13019191296)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 0ns
           - MaxCompressedTextFileLength: 0
           - NumDisksAccessed: 0
           - NumScannerThreadsStarted: 35
           - PeakMemoryUsage: 1.12 GB (1204580352)
           - PerReadThreadRawHdfsThroughput: 199.17 MB/sec
           - RemoteScanRanges: 0
           - RowsRead: 24.20M (24203155)
           - RowsReturned: 24.14M (24136595)
           - RowsReturnedRate: 52.68 M/sec
           - ScanRangesComplete: 70
           - ScannerThreadsInvoluntaryContextSwitches: 0
           - ScannerThreadsTotalWallClockTime: 0ns
             - DelimiterParseTime: 36s041ms
             - MaterializeTupleTime(*): 1m17s
             - ScannerThreadsSysTime: 0ns
             - ScannerThreadsUserTime: 0ns
           - ScannerThreadsVoluntaryContextSwitches: 0
           - TotalRawHdfsReadTime(*): 1m2s
           - TotalReadThroughput: 36.84 MB/sec
      Instance 6b417e40e3a54863:84ca6e5163f057ac (host=node03:22000):(Total: 5m33s, non-child: 0ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2739/342.31 GB 
        MemoryUsage(8s000ms): 937.27 MB, 2.87 GB, 2.70 GB, 2.75 GB, 2.79 GB, 2.84 GB, 2.91 GB, 2.96 GB, 3.06 GB, 3.09 GB, 3.13 GB, 3.18 GB, 3.30 GB, 3.46 GB, 3.50 GB, 3.48 GB, 3.28 GB, 3.13 GB, 3.57 GB, 3.71 GB, 3.73 GB, 3.79 GB, 3.86 GB, 3.92 GB, 3.99 GB, 4.07 GB, 4.12 GB, 4.21 GB, 4.33 GB, 4.28 GB, 4.12 GB, 4.62 GB, 4.72 GB, 4.72 GB, 4.77 GB, 4.91 GB, 5.05 GB, 5.17 GB, 5.21 GB, 5.24 GB, 5.31 GB
        ThreadUsage(8s000ms): 27, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36
         - AverageThreadTokens: 35.79 
         - PeakMemoryUsage: 5.40 GB (5795635200)
         - PerHostPeakMemUsage: 5.40 GB (5795635200)
         - PrepareTime: 194.134ms
         - RowsProduced: 26.14M (26144270)
         - TotalCpuTime: 0ns
         - TotalNetworkReceiveTime: 0ns
         - TotalNetworkSendTime: 0ns
         - TotalStorageWaitTime: 1m1s
        BlockMgr:
           - BlockWritesOutstanding: 0
           - BlocksCreated: 0
           - BlocksRecycled: 0
           - BufferedPins: 0
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 43.20 GB (46385647616)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0ns
           - TotalEncryptionTime: 0ns
           - TotalIntegrityCheckTime: 0ns
           - TotalReadBlockTime: 0ns
        CodeGen:(Total: 2s115ms, non-child: 2s115ms, % non-child: 100.00%)
           - CodegenTime: 25.10ms
           - CompileTime: 681.155ms
           - LoadTime: 23.82us
           - ModuleFileSize: 1.98 MB (2073272)
           - OptimizationTime: 1s263ms
           - PrepareTime: 169.734ms
        HdfsTableSink:(Total: 5m32s, non-child: 5m32s, % non-child: 100.00%)
           - BytesWritten: 456.00 B (456)
           - CompressTimer: 12s968ms
           - EncodeTimer: 4m55s
           - FilesCreated: 114
           - HdfsWriteTimer: 0ns
           - PartitionsCreated: 114
           - PeakMemoryUsage: 4.48 GB (4810268672)
           - RowsInserted: 0
           - TmpFileCreateTimer: 1s282ms
        HDFS_SCAN_NODE (id=0):(Total: 542.747ms, non-child: 542.747ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2739/342.31 GB 
          BytesRead(8s000ms): 352.50 MB, 1.12 GB, 1.16 GB, 1.40 GB, 1.62 GB, 1.85 GB, 2.10 GB, 2.35 GB, 2.60 GB, 2.85 GB, 3.11 GB, 3.37 GB, 3.63 GB, 3.88 GB, 4.13 GB, 4.35 GB, 4.39 GB, 4.48 GB, 5.16 GB, 5.53 GB, 5.80 GB, 6.13 GB, 6.46 GB, 6.79 GB, 7.13 GB, 7.48 GB, 7.83 GB, 8.18 GB, 8.52 GB, 8.75 GB, 8.86 GB, 9.59 GB, 9.95 GB, 10.27 GB, 10.60 GB, 10.93 GB, 11.26 GB, 11.59 GB, 11.93 GB, 12.26 GB, 12.59 GB
           - AverageHdfsReadThreadConcurrency: 0.28 
           - AverageScannerThreadConcurrency: 35.00 
           - BytesRead: 13.05 GB (14009047040)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 0ns
           - MaxCompressedTextFileLength: 0
           - NumDisksAccessed: 0
           - NumScannerThreadsStarted: 35
           - PeakMemoryUsage: 1.14 GB (1220251648)
           - PerReadThreadRawHdfsThroughput: 144.33 MB/sec
           - RemoteScanRanges: 0
           - RowsRead: 26.21M (26209806)
           - RowsReturned: 26.14M (26144270)
           - RowsReturnedRate: 48.17 M/sec
           - ScanRangesComplete: 70
           - ScannerThreadsInvoluntaryContextSwitches: 0
           - ScannerThreadsTotalWallClockTime: 0ns
             - DelimiterParseTime: 38s958ms
             - MaterializeTupleTime(*): 1m25s
             - ScannerThreadsSysTime: 0ns
             - ScannerThreadsUserTime: 0ns
           - ScannerThreadsVoluntaryContextSwitches: 0
           - TotalRawHdfsReadTime(*): 1m32s
           - TotalReadThroughput: 39.81 MB/sec