Query (id=f04dba71f0b5a296:f4f5392c0729d4b3)
  Summary
    Session ID: c34c85cdc96c7516:4527c7e5aef835b0
    Session Type: BEESWAX
    Start Time: 2015-01-21 13:59:24.958859000
    End Time: 2015-01-21 14:00:58.597579000
    Query Type: DDL
    Query State: EXCEPTION
    Query Status: Memory limit exceeded
    Impala Version: impalad version 1.4.2-cdh5 RELEASE (build eac952d4ff674663ec3834778c2b981b252aec78)
    User: hdpdev
    Connected User: hdpdev
    Delegated User: 
    Network Address: 10.30.30.47:59714
    Default Db: default
    Sql Statement: create TABLE dev_temp.load_prd_src_report_step0_1 STORED AS parquet  AS   select c.event_date,c.event_85_id,c.dw_websession_id,c.is_signup,c.request_url_query,c.request_url_stem,a.query,a.url from dev_temp.load_prd_src_report_step0 c  join dev_ods.referrer a on c.referrer_id=a.referrer_id
    DDL Type: CREATE_TABLE_AS_SELECT
    Plan: 
----------------
Estimated Per-Host Requirements: Memory=48.52GB VCores=2

F00:PLAN FRAGMENT [RANDOM]
  WRITE TO HDFS [dev_temp.load_prd_src_report_step0_1, OVERWRITE=false]
  |  partitions=1
  |  hosts=4 per-host-mem=1.00GB
  |
  02:HASH JOIN [INNER JOIN, BROADCAST]
  |  hash predicates: a.referrer_id = c.referrer_id
  |  hosts=4 per-host-mem=47.26GB
  |  tuple-ids=3,2 row-size=381B cardinality=1590394716
  |
  |--03:EXCHANGE [BROADCAST]
  |     hosts=4 per-host-mem=0B
  |     tuple-ids=2 row-size=152B cardinality=303077754
  |
  01:SCAN HDFS [dev_ods.referrer a, RANDOM]
     partitions=1/1 size=56.10GB
     table stats: 631892075 rows total
     column stats: all
     hosts=4 per-host-mem=264.00MB
     tuple-ids=3 row-size=229B cardinality=631892075

F01:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F00, EXCHANGE=03, BROADCAST]
  00:SCAN HDFS [dev_temp.load_prd_src_report_step0 c, RANDOM]
     partitions=1/1 size=12.06GB
     table stats: 303077754 rows total
     column stats: all
     hosts=4 per-host-mem=616.00MB
     tuple-ids=2 row-size=152B cardinality=303077754
----------------
    Estimated Per-Host Mem: 52095012045
    Estimated Per-Host VCores: 2
    Admission result: Admitted immediately
    Request Pool: root.hdpdev
    ExecSummary: 
Operator          #Hosts  Avg Time  Max Time    #Rows  Est. #Rows  Peak Mem  Est. Peak Mem  Detail                         
---------------------------------------------------------------------------------------------------------------------------
02:HASH JOIN           4       0ns       0ns        0       1.59B  27.59 GB       47.26 GB  INNER JOIN, BROADCAST          
|--03:EXCHANGE         4       0ns       0ns  149.72M     303.08M         0              0  BROADCAST                      
|  00:SCAN HDFS        4   1s657ms   2s354ms  145.67M     303.08M   1.75 GB      616.00 MB  dev_temp.load_prd_src_repor... 
01:SCAN HDFS           4       0ns       0ns        0     631.89M         0      264.00 MB  dev_ods.referrer a             
    Query Timeline
      Start execution: 100773
      Planning finished: 58014684
      Submit for admission: 103060466
      Completed admission: 103220977
      Rows available: 92886622252
      Unregister query: 93638708399
  ImpalaServer
    - AsyncTotalTime: 0
    - ClientFetchWaitTimer: 0
    - InactiveTotalTime: 0
    - RowMaterializationTimer: 0
    - TotalTime: 0
  Execution Profile f04dba71f0b5a296:f4f5392c0729d4b3
    Per Node Peak Memory Usage: x.xdn5.x.x:22000(27.33 GB) x.xdn8.x.x:22000(27.59 GB) x.xdn3.x.x:22000(26.31 GB) x.xdn1.x.x:22000(26.99 GB) 
    - AsyncTotalTime: 0
    - FinalizationTimer: 854490
    - InactiveTotalTime: 0
    - TotalTime: 92779961133
    Averaged Fragment F00
      split sizes:  min: 13.92 GB, max: 14.13 GB, avg: 14.02 GB, stddev: 85.62 MB
      completion times: min:1m32s  max:1m32s  mean: 1m32s  stddev:1.801ms
      execution rates: min:153.70 MB/sec  max:156.07 MB/sec  mean:154.85 MB/sec  stddev:945.52 KB/sec
      num instances: 4
      - AsyncTotalTime: 0
      - AverageThreadTokens: 14.46989212478344
      - InactiveTotalTime: 0
      - PeakMemoryUsage: 29047357836
      - PrepareTime: 3891436
      - RowsProduced: 0
      - TotalCpuTime: 273041605077
      - TotalNetworkReceiveTime: 44220688782
      - TotalNetworkSendTime: 0
      - TotalStorageWaitTime: 0
      - TotalTime: 0
      CodeGen
        - AsyncTotalTime: 0
        - CodegenTime: 134179
        - CompileTime: 53559145
        - InactiveTotalTime: 0
        - LoadTime: 36542143
        - ModuleFileSize: 369500
        - TotalTime: 91079061
      HdfsTableSink
        - AsyncTotalTime: 0
        - BytesWritten: 0
        - EncodeTimer: 0
        - HdfsWriteTimer: 0
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 0
        - RowsInserted: 0
        - TotalTime: 39923
      HASH_JOIN_NODE (id=2)
        - AsyncTotalTime: 0
        - BuildBuckets: 134217728
        - BuildRows: 146067712
        - BuildTime: 42636364991
        - InactiveTotalTime: 0
        - LeftChildRows: 0
        - LeftChildTime: 0
        - LoadFactor: 0.55164735019207
        - PeakMemoryUsage: 29041753394
        - RowsReturned: 0
        - RowsReturnedRate: 0
        - TotalTime: 389920
        EXCHANGE_NODE (id=3)
          - AsyncTotalTime: 0
          - BytesReceived: 11593198942
          - ConvertRowBatchTime: 2802555011
          - DeserializeRowBatchTimer: 51565756195
          - FirstBatchArrivalWaitTime: 61414585
          - InactiveTotalTime: 0
          - PeakMemoryUsage: 0
          - RowsReturned: 146067712
          - RowsReturnedRate: 3085201
          - SendersBlockedTimer: 27423177405
          - SendersBlockedTotalTimer(*): 88393974454
          - TotalTime: 47365348466
        HDFS_SCAN_NODE (id=1)
          - AsyncTotalTime: 0
          - AverageHdfsReadThreadConcurrency: 0.0
          - AverageScannerThreadConcurrency: 12.501565666765933
          - BytesRead: 0
          - BytesReadDataNodeCache: 0
          - BytesReadLocal: 0
          - BytesReadShortCircuit: 0
          - InactiveTotalTime: 0
          - NumDisksAccessed: 0
          - NumScannerThreadsStarted: 38
          - PeakMemoryUsage: 0
          - PerReadThreadRawHdfsThroughput: 0
          - RowsRead: 0
          - RowsReturned: 0
          - RowsReturnedRate: 0
          - ScanRangesComplete: 0
          - ScannerThreadsInvoluntaryContextSwitches: 38914
          - ScannerThreadsTotalWallClockTime: 273041660069
            - MaterializeTupleTime(*): 0
            - ScannerThreadsSysTime: 241841972250
            - ScannerThreadsUserTime: 22857513750
          - ScannerThreadsVoluntaryContextSwitches: 2624660
          - TotalRawHdfsReadTime(*): 0
          - TotalReadThroughput: 0
          - TotalTime: 29060
    Averaged Fragment F01
      split sizes:  min: 2.86 GB, max: 3.13 GB, avg: 3.01 GB, stddev: 99.44 MB
      completion times: min:1m32s  max:1m32s  mean: 1m32s  stddev:1.291ms
      execution rates: min:31.62 MB/sec  max:34.54 MB/sec  mean:33.30 MB/sec  stddev:1.07 MB/sec
      num instances: 4
      - AsyncTotalTime: 0
      - AverageThreadTokens: 9.975131289692234
      - InactiveTotalTime: 0
      - PeakMemoryUsage: 1845033790
      - PrepareTime: 105703
      - RowsProduced: 36418560
      - TotalCpuTime: 0
      - TotalNetworkReceiveTime: 0
      - TotalNetworkSendTime: 51928248576
      - TotalStorageWaitTime: 380014164
      - TotalTime: 89839784534
      CodeGen
        - AsyncTotalTime: 0
        - CodegenTime: 0
        - CompileTime: 26584306
        - InactiveTotalTime: 0
        - LoadTime: 31736632
        - ModuleFileSize: 369500
        - TotalTime: 59125155
      DataStreamSender (dst_id=3)
        - AsyncTotalTime: 0
        - BytesSent: 2890021156
        - InactiveTotalTime: 0
        - NetworkThroughput(*): 11909694
        - OverallThroughput: 32781514
        - PeakMemoryUsage: 7264
        - SerializeBatchTime: 35546265388
        - ThriftTransmitTime(*): 243173068924
        - TotalTime: 88120278381
        - UncompressedRowBatchSize: 5943875744
      HDFS_SCAN_NODE (id=0)
        - AsyncTotalTime: 0
        - AverageHdfsReadThreadConcurrency: 0.008421564774023792
        - AverageScannerThreadConcurrency: 9.0
        - BytesRead: 2318067467
        - BytesReadDataNodeCache: 0
        - BytesReadLocal: 0
        - BytesReadShortCircuit: 0
        - DecompressionTime: 4291332365
        - InactiveTotalTime: 0
        - NumColumns: 0
        - NumDisksAccessed: 0
        - NumScannerThreadsStarted: 9
        - PeakMemoryUsage: 1844834066
        - PerReadThreadRawHdfsThroughput: 1927514885
        - RowsRead: 36578560
        - RowsReturned: 36418560
        - RowsReturnedRate: 25895098
        - ScanRangesComplete: 0
        - ScannerThreadsInvoluntaryContextSwitches: 0
        - ScannerThreadsTotalWallClockTime: 0
          - MaterializeTupleTime(*): 0
          - ScannerThreadsSysTime: 0
          - ScannerThreadsUserTime: 0
        - ScannerThreadsVoluntaryContextSwitches: 0
        - TotalRawHdfsReadTime(*): 1208719282
        - TotalReadThroughput: 25752548
        - TotalTime: 1657090466
    Fragment F00
      Instance f04dba71f0b5a296:f4f5392c0729d4b6 (host=x.xdn1.x.x:22000)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/1.93 GB 1:5/1.61 GB 2:2/603.81 MB 3:6/1.64 GB 4:3/989.63 MB 5:4/1.29 GB 6:4/1.28 GB 7:1/329.84 MB 8:4/1.29 GB 9:1/329.04 MB 10:2/658.88 MB 11:7/2.25 GB 
        - AsyncTotalTime: 0
        - AverageThreadTokens: 11.055555555555555
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 28978130856
        - PrepareTime: 3868114
        - RowsProduced: 0
        - TotalCpuTime: 91496096572
        - TotalNetworkReceiveTime: 43572533632
        - TotalNetworkSendTime: 0
        - TotalStorageWaitTime: 0
        - TotalTime: 0
        CodeGen
          - AsyncTotalTime: 0
          - CodegenTime: 110572
          - CompileTime: 55545914
          - InactiveTotalTime: 0
          - LoadTime: 34735363
          - ModuleFileSize: 369500
          - TotalTime: 91181376
        HdfsTableSink
          - AsyncTotalTime: 0
          - BytesWritten: 0
          - EncodeTimer: 0
          - HdfsWriteTimer: 0
          - InactiveTotalTime: 0
          - PeakMemoryUsage: 0
          - RowsInserted: 0
          - TotalTime: 35865
        HASH_JOIN_NODE (id=2)
          ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
          - AsyncTotalTime: 0
          - BuildBuckets: 134217728
          - BuildRows: 145668096
          - BuildTime: 43187741182
          - InactiveTotalTime: 0
          - LeftChildRows: 0
          - LeftChildTime: 0
          - LoadFactor: 0.5505366921424866
          - PeakMemoryUsage: 28978098088
          - RowsReturned: 0
          - RowsReturnedRate: 0
          - TotalTime: 385427
          EXCHANGE_NODE (id=3)
            - AsyncTotalTime: 0
            - BytesReceived: 11558691028
            - ConvertRowBatchTime: 2678413912
            - DeserializeRowBatchTimer: 50531006811
            - FirstBatchArrivalWaitTime: 61172551
            - InactiveTotalTime: 7254248290
            - PeakMemoryUsage: 0
            - RowsReturned: 145668096
            - RowsReturnedRate: 3127903
            - SendersBlockedTimer: 25107310431
            - SendersBlockedTotalTimer(*): 83111399075
            - TotalTime: 46570524279
          HDFS_SCAN_NODE (id=1)
            Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/1.93 GB 1:5/1.61 GB 2:2/603.81 MB 3:6/1.64 GB 4:3/989.63 MB 5:4/1.29 GB 6:4/1.28 GB 7:1/329.84 MB 8:4/1.29 GB 9:1/329.04 MB 10:2/658.88 MB 11:7/2.25 GB 
            - AsyncTotalTime: 0
            - AverageHdfsReadThreadConcurrency: 0.0
            - AverageScannerThreadConcurrency: 9.111731843575418
            - BytesRead: 0
            - BytesReadDataNodeCache: 0
            - BytesReadLocal: 0
            - BytesReadShortCircuit: 0
            - InactiveTotalTime: 0
            - NumDisksAccessed: 0
            - NumScannerThreadsStarted: 27
            - PeakMemoryUsage: 0
            - PerReadThreadRawHdfsThroughput: 0
            - RowsRead: 0
            - RowsReturned: 0
            - RowsReturnedRate: 0
            - ScanRangesComplete: 0
            - ScannerThreadsInvoluntaryContextSwitches: 11801
            - ScannerThreadsTotalWallClockTime: 91496135661
              - MaterializeTupleTime(*): 0
              - ScannerThreadsSysTime: 77131265000
              - ScannerThreadsUserTime: 11578232000
            - ScannerThreadsVoluntaryContextSwitches: 1142975
            - TotalRawHdfsReadTime(*): 0
            - TotalReadThroughput: 0
            - TotalTime: 26410
      Instance f04dba71f0b5a296:f4f5392c0729d4b4 (host=x.xdn5.x.x:22000)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/1.93 GB 1:4/1.29 GB 2:6/1.93 GB 4:3/988.03 MB 5:3/987.94 MB 6:2/659.69 MB 7:3/904.62 MB 8:2/659.86 MB 9:4/1.29 GB 10:5/1.61 GB 11:6/1.93 GB 
        - AsyncTotalTime: 0
        - AverageThreadTokens: 13.763736263736265
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 29341605400
        - PrepareTime: 3912450
        - RowsProduced: 0
        - TotalCpuTime: 99811211446
        - TotalNetworkReceiveTime: 46308172158
        - TotalNetworkSendTime: 0
        - TotalStorageWaitTime: 0
        - TotalTime: 0
        CodeGen
          - AsyncTotalTime: 0
          - CodegenTime: 121927
          - CompileTime: 52730931
          - InactiveTotalTime: 0
          - LoadTime: 39345277
          - ModuleFileSize: 369500
          - TotalTime: 93068485
        HdfsTableSink
          - AsyncTotalTime: 0
          - BytesWritten: 0
          - EncodeTimer: 0
          - HdfsWriteTimer: 0
          - InactiveTotalTime: 0
          - PeakMemoryUsage: 0
          - RowsInserted: 0
          - TotalTime: 46195
        HASH_JOIN_NODE (id=2)
          ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
          - AsyncTotalTime: 0
          - BuildBuckets: 134217728
          - BuildRows: 147951616
          - BuildTime: 41221087335
          - InactiveTotalTime: 0
          - LeftChildRows: 0
          - LeftChildTime: 0
          - LoadFactor: 0.5571830868721008
          - PeakMemoryUsage: 29341564440
          - RowsReturned: 0
          - RowsReturnedRate: 0
          - TotalTime: 371457
          EXCHANGE_NODE (id=3)
            - AsyncTotalTime: 0
            - BytesReceived: 11740444296
            - ConvertRowBatchTime: 2880933261
            - DeserializeRowBatchTimer: 52255592130
            - FirstBatchArrivalWaitTime: 59539687
            - InactiveTotalTime: 9690108484
            - PeakMemoryUsage: 0
            - RowsReturned: 147951616
            - RowsReturnedRate: 2986903
            - SendersBlockedTimer: 26809300205
            - SendersBlockedTotalTimer(*): 88267294088
            - TotalTime: 49533436951
          HDFS_SCAN_NODE (id=1)
            Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/1.93 GB 1:4/1.29 GB 2:6/1.93 GB 4:3/988.03 MB 5:3/987.94 MB 6:2/659.69 MB 7:3/904.62 MB 8:2/659.86 MB 9:4/1.29 GB 10:5/1.61 GB 11:6/1.93 GB 
            - AsyncTotalTime: 0
            - AverageHdfsReadThreadConcurrency: 0.0
            - AverageScannerThreadConcurrency: 11.834254143646408
            - BytesRead: 0
            - BytesReadDataNodeCache: 0
            - BytesReadLocal: 0
            - BytesReadShortCircuit: 0
            - InactiveTotalTime: 0
            - NumDisksAccessed: 0
            - NumScannerThreadsStarted: 29
            - PeakMemoryUsage: 0
            - PerReadThreadRawHdfsThroughput: 0
            - RowsRead: 0
            - RowsReturned: 0
            - RowsReturnedRate: 0
            - ScanRangesComplete: 0
            - ScannerThreadsInvoluntaryContextSwitches: 12589
            - ScannerThreadsTotalWallClockTime: 99811257687
              - MaterializeTupleTime(*): 0
              - ScannerThreadsSysTime: 86795797000
              - ScannerThreadsUserTime: 9912489000
            - ScannerThreadsVoluntaryContextSwitches: 1139317
            - TotalRawHdfsReadTime(*): 0
            - TotalReadThroughput: 0
            - TotalTime: 27334
      Instance f04dba71f0b5a296:f4f5392c0729d4b5 (host=x.xdn8.x.x:22000)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:5/1.61 GB 1:4/1.29 GB 2:4/1.29 GB 3:5/1.61 GB 4:3/688.34 MB 5:4/1.29 GB 6:4/1.07 GB 7:5/1.61 GB 8:2/660.04 MB 9:7/2.25 GB 10:2/660.40 MB 
        - AsyncTotalTime: 0
        - AverageThreadTokens: 16.54891304347826
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 29623718744
        - PrepareTime: 3871241
        - RowsProduced: 0
        - TotalCpuTime: 468422552785
        - TotalNetworkReceiveTime: 44636854584
        - TotalNetworkSendTime: 0
        - TotalStorageWaitTime: 0
        - TotalTime: 0
        CodeGen
          - AsyncTotalTime: 0
          - CodegenTime: 152408
          - CompileTime: 52647121
          - InactiveTotalTime: 0
          - LoadTime: 36729997
          - ModuleFileSize: 369500
          - TotalTime: 90361847
        HdfsTableSink
          - AsyncTotalTime: 0
          - BytesWritten: 0
          - EncodeTimer: 0
          - HdfsWriteTimer: 0
          - InactiveTotalTime: 0
          - PeakMemoryUsage: 0
          - RowsInserted: 0
          - TotalTime: 38873
        HASH_JOIN_NODE (id=2)
          ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
          - AsyncTotalTime: 0
          - BuildBuckets: 134217728
          - BuildRows: 149721088
          - BuildTime: 43826407634
          - InactiveTotalTime: 0
          - LeftChildRows: 0
          - LeftChildTime: 0
          - LoadFactor: 0.5623002648353577
          - PeakMemoryUsage: 29623346552
          - RowsReturned: 0
          - RowsReturnedRate: 0
          - TotalTime: 394093
          EXCHANGE_NODE (id=3)
            - AsyncTotalTime: 0
            - BytesReceived: 11881854594
            - ConvertRowBatchTime: 2913481019
            - DeserializeRowBatchTimer: 53335381674
            - FirstBatchArrivalWaitTime: 61932238
            - InactiveTotalTime: 7499414310
            - PeakMemoryUsage: 0
            - RowsReturned: 149721088
            - RowsReturnedRate: 3124948
            - SendersBlockedTimer: 29641800471
            - SendersBlockedTotalTimer(*): 92912743228
            - TotalTime: 47911539947
          HDFS_SCAN_NODE (id=1)
            Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:5/1.61 GB 1:4/1.29 GB 2:4/1.29 GB 3:5/1.61 GB 4:3/688.34 MB 5:4/1.29 GB 6:4/1.07 GB 7:5/1.61 GB 8:2/660.04 MB 9:7/2.25 GB 10:2/660.40 MB 
            - AsyncTotalTime: 0
            - AverageHdfsReadThreadConcurrency: 0.0
            - AverageScannerThreadConcurrency: 14.548913043478262
            - BytesRead: 0
            - BytesReadDataNodeCache: 0
            - BytesReadLocal: 0
            - BytesReadShortCircuit: 0
            - InactiveTotalTime: 0
            - NumDisksAccessed: 0
            - NumScannerThreadsStarted: 48
            - PeakMemoryUsage: 0
            - PerReadThreadRawHdfsThroughput: 0
            - RowsRead: 0
            - RowsReturned: 0
            - RowsReturnedRate: 0
            - ScanRangesComplete: 0
            - ScannerThreadsInvoluntaryContextSwitches: 68522
            - ScannerThreadsTotalWallClockTime: 468422613643
              - MaterializeTupleTime(*): 0
              - ScannerThreadsSysTime: 416993591000
              - ScannerThreadsUserTime: 37475286000
            - ScannerThreadsVoluntaryContextSwitches: 4359404
            - TotalRawHdfsReadTime(*): 0
            - TotalReadThroughput: 0
            - TotalTime: 29201
      Instance f04dba71f0b5a296:f4f5392c0729d4b7 (host=x.xdn3.x.x:22000)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:5/1.61 GB 1:4/1.29 GB 2:5/1.61 GB 3:6/1.93 GB 4:4/1.29 GB 5:4/997.97 MB 6:2/659.15 MB 7:2/660.01 MB 8:2/396.05 MB 9:3/988.78 MB 10:5/1.61 GB 11:3/989.12 MB 
        - AsyncTotalTime: 0
        - AverageThreadTokens: 16.511363636363637
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 28245976344
        - PrepareTime: 3913940
        - RowsProduced: 0
        - TotalCpuTime: 432436559506
        - TotalNetworkReceiveTime: 42365194756
        - TotalNetworkSendTime: 0
        - TotalStorageWaitTime: 0
        - TotalTime: 0
        CodeGen
          - AsyncTotalTime: 0
          - CodegenTime: 151810
          - CompileTime: 53312615
          - InactiveTotalTime: 0
          - LoadTime: 35357936
          - ModuleFileSize: 369500
          - TotalTime: 89704537
        HdfsTableSink
          - AsyncTotalTime: 0
          - BytesWritten: 0
          - EncodeTimer: 0
          - HdfsWriteTimer: 0
          - InactiveTotalTime: 0
          - PeakMemoryUsage: 0
          - RowsInserted: 0
          - TotalTime: 38762
        HASH_JOIN_NODE (id=2)
          ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
          - AsyncTotalTime: 0
          - BuildBuckets: 134217728
          - BuildRows: 140930048
          - BuildTime: 42310223816
          - InactiveTotalTime: 0
          - LeftChildRows: 0
          - LeftChildTime: 0
          - LoadFactor: 0.536569356918335
          - PeakMemoryUsage: 28224004496
          - RowsReturned: 0
          - RowsReturnedRate: 0
          - TotalTime: 408704
          EXCHANGE_NODE (id=3)
            - AsyncTotalTime: 0
            - BytesReceived: 11191805851
            - ConvertRowBatchTime: 2737391853
            - DeserializeRowBatchTimer: 50141044165
            - FirstBatchArrivalWaitTime: 63013865
            - InactiveTotalTime: 6596215752
            - PeakMemoryUsage: 0
            - RowsReturned: 140930048
            - RowsReturnedRate: 3101051
            - SendersBlockedTimer: 28134298514
            - SendersBlockedTotalTimer(*): 89284461428
            - TotalTime: 45445892689
          HDFS_SCAN_NODE (id=1)
            Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:5/1.61 GB 1:4/1.29 GB 2:5/1.61 GB 3:6/1.93 GB 4:4/1.29 GB 5:4/997.97 MB 6:2/659.15 MB 7:2/660.01 MB 8:2/396.05 MB 9:3/988.78 MB 10:5/1.61 GB 11:3/989.12 MB 
            - AsyncTotalTime: 0
            - AverageHdfsReadThreadConcurrency: 0.0
            - AverageScannerThreadConcurrency: 14.511363636363637
            - BytesRead: 0
            - BytesReadDataNodeCache: 0
            - BytesReadLocal: 0
            - BytesReadShortCircuit: 0
            - InactiveTotalTime: 0
            - NumDisksAccessed: 0
            - NumScannerThreadsStarted: 48
            - PeakMemoryUsage: 0
            - PerReadThreadRawHdfsThroughput: 0
            - RowsRead: 0
            - RowsReturned: 0
            - RowsReturnedRate: 0
            - ScanRangesComplete: 0
            - ScannerThreadsInvoluntaryContextSwitches: 62745
            - ScannerThreadsTotalWallClockTime: 432436633285
              - MaterializeTupleTime(*): 0
              - ScannerThreadsSysTime: 386447236000
              - ScannerThreadsUserTime: 32464048000
            - ScannerThreadsVoluntaryContextSwitches: 3856945
            - TotalRawHdfsReadTime(*): 0
            - TotalReadThroughput: 0
            - TotalTime: 33296
    Fragment F01
      Instance f04dba71f0b5a296:f4f5392c0729d4b8 (host=x.xdn1.x.x:22000)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:3/998.37 MB 3:1/355.81 MB 4:1/355.81 MB 7:1/355.63 MB 9:2/711.71 MB 10:1/355.70 MB 
        - AsyncTotalTime: 0
        - AverageThreadTokens: 9.951086956521738
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 1858599560
        - PrepareTime: 105625
        - RowsProduced: 40931328
        - TotalCpuTime: 0
        - TotalNetworkReceiveTime: 0
        - TotalNetworkSendTime: 51720085042
        - TotalStorageWaitTime: 390831446
        - TotalTime: 91839514605
        CodeGen
          - AsyncTotalTime: 0
          - CodegenTime: 0
          - CompileTime: 26568278
          - InactiveTotalTime: 0
          - LoadTime: 33198381
          - ModuleFileSize: 369500
          - TotalTime: 60581076
        DataStreamSender (dst_id=3)
          - AsyncTotalTime: 0
          - BytesSent: 3263747181
          - InactiveTotalTime: 0
          - NetworkThroughput(*): 12395528
          - OverallThroughput: 35898479
          - PeakMemoryUsage: 7264
          - SerializeBatchTime: 38649678939
          - ThriftTransmitTime(*): 263300372036
          - TotalTime: 90916027356
          - UncompressedRowBatchSize: 6680778333
        HDFS_SCAN_NODE (id=0)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:3/998.37 MB 3:1/355.81 MB 4:1/355.81 MB 7:1/355.63 MB 9:2/711.71 MB 10:1/355.70 MB 
          - AsyncTotalTime: 0
          - AverageHdfsReadThreadConcurrency: 0.00546448087431694
          - AverageScannerThreadConcurrency: 9.0
          - BytesRead: 2521315936
          - BytesReadDataNodeCache: 0
          - BytesReadLocal: 0
          - BytesReadShortCircuit: 0
          - DecompressionTime: 4710322346
          - InactiveTotalTime: 0
          - NumColumns: 0
          - NumDisksAccessed: 0
          - NumScannerThreadsStarted: 9
          - PeakMemoryUsage: 1858526752
          - PerReadThreadRawHdfsThroughput: 1996917456
          - RowsRead: 41094144
          - RowsReturned: 40931328
          - RowsReturnedRate: 47515040
          - ScanRangesComplete: 0
          - ScannerThreadsInvoluntaryContextSwitches: 0
          - ScannerThreadsTotalWallClockTime: 0
            - MaterializeTupleTime(*): 0
            - ScannerThreadsSysTime: 0
            - ScannerThreadsUserTime: 0
          - ScannerThreadsVoluntaryContextSwitches: 0
          - TotalRawHdfsReadTime(*): 1262603984
          - TotalReadThroughput: 27405608
          - TotalTime: 861439400
      Instance f04dba71f0b5a296:f4f5392c0729d4b9 (host=x.xdn8.x.x:22000)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/711.63 MB 2:3/944.87 MB 3:1/355.71 MB 7:2/711.53 MB 10:1/355.75 MB 
        - AsyncTotalTime: 0
        - AverageThreadTokens: 10.0
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 1863552248
        - PrepareTime: 96729
        - RowsProduced: 33081344
        - TotalCpuTime: 0
        - TotalNetworkReceiveTime: 0
        - TotalNetworkSendTime: 55508708686
        - TotalStorageWaitTime: 565762055
        - TotalTime: 90854268351
        CodeGen
          - AsyncTotalTime: 0
          - CodegenTime: 0
          - CompileTime: 26382905
          - InactiveTotalTime: 0
          - LoadTime: 30000475
          - ModuleFileSize: 369500
          - TotalTime: 57186417
        DataStreamSender (dst_id=3)
          - AsyncTotalTime: 0
          - BytesSent: 2628176553
          - InactiveTotalTime: 0
          - NetworkThroughput(*): 10580947
          - OverallThroughput: 29551834
          - PeakMemoryUsage: 7264
          - SerializeBatchTime: 32683442117
          - ThriftTransmitTime(*): 248387643436
          - TotalTime: 88934463557
          - UncompressedRowBatchSize: 5399547199
        HDFS_SCAN_NODE (id=0)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:2/711.63 MB 2:3/944.87 MB 3:1/355.71 MB 7:2/711.53 MB 10:1/355.75 MB 
          - AsyncTotalTime: 0
          - AverageHdfsReadThreadConcurrency: 0.005494505494505495
          - AverageScannerThreadConcurrency: 9.0
          - BytesRead: 2204169842
          - BytesReadDataNodeCache: 0
          - BytesReadLocal: 0
          - BytesReadShortCircuit: 0
          - DecompressionTime: 3863275658
          - InactiveTotalTime: 0
          - NumColumns: 0
          - NumDisksAccessed: 0
          - NumScannerThreadsStarted: 9
          - PeakMemoryUsage: 1863266472
          - PerReadThreadRawHdfsThroughput: 2127151434
          - RowsRead: 33232896
          - RowsReturned: 33081344
          - RowsReturnedRate: 17775339
          - ScanRangesComplete: 0
          - ScannerThreadsInvoluntaryContextSwitches: 0
          - ScannerThreadsTotalWallClockTime: 0
            - MaterializeTupleTime(*): 0
            - ScannerThreadsSysTime: 0
            - ScannerThreadsUserTime: 0
          - ScannerThreadsVoluntaryContextSwitches: 0
          - TotalRawHdfsReadTime(*): 1036207299
          - TotalReadThroughput: 24221646
          - TotalTime: 1861080886
      Instance f04dba71f0b5a296:f4f5392c0729d4ba (host=x.xdn5.x.x:22000)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:2/600.37 MB 2:1/355.71 MB 3:1/197.01 MB 4:1/355.69 MB 5:1/355.84 MB 7:1/355.66 MB 8:1/355.60 MB 10:1/355.67 MB 
        - AsyncTotalTime: 0
        - AverageThreadTokens: 9.94943820224719
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 1778259904
        - PrepareTime: 112131
        - RowsProduced: 35719168
        - TotalCpuTime: 0
        - TotalNetworkReceiveTime: 0
        - TotalNetworkSendTime: 51034449370
        - TotalStorageWaitTime: 267715849
        - TotalTime: 88828317837
        CodeGen
          - AsyncTotalTime: 0
          - CodegenTime: 0
          - CompileTime: 27208705
          - InactiveTotalTime: 0
          - LoadTime: 32655538
          - ModuleFileSize: 369500
          - TotalTime: 60632469
        DataStreamSender (dst_id=3)
          - AsyncTotalTime: 0
          - BytesSent: 2831757302
          - InactiveTotalTime: 0
          - NetworkThroughput(*): 11660684
          - OverallThroughput: 32469397
          - PeakMemoryUsage: 7264
          - SerializeBatchTime: 35561222456
          - ThriftTransmitTime(*): 242846570916
          - TotalTime: 87213114995
          - UncompressedRowBatchSize: 5829360334
        HDFS_SCAN_NODE (id=0)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:2/600.37 MB 2:1/355.71 MB 3:1/197.01 MB 4:1/355.69 MB 5:1/355.84 MB 7:1/355.66 MB 8:1/355.60 MB 10:1/355.67 MB 
          - AsyncTotalTime: 0
          - AverageHdfsReadThreadConcurrency: 0.0
          - AverageScannerThreadConcurrency: 9.0
          - BytesRead: 2232591725
          - BytesReadDataNodeCache: 0
          - BytesReadLocal: 0
          - BytesReadShortCircuit: 0
          - DecompressionTime: 4456840288
          - InactiveTotalTime: 0
          - NumColumns: 0
          - NumDisksAccessed: 0
          - NumScannerThreadsStarted: 9
          - PeakMemoryUsage: 1777974304
          - PerReadThreadRawHdfsThroughput: 1799721683
          - RowsRead: 35881984
          - RowsReturned: 35719168
          - RowsReturnedRate: 23024540
          - ScanRangesComplete: 0
          - ScannerThreadsInvoluntaryContextSwitches: 0
          - ScannerThreadsTotalWallClockTime: 0
            - MaterializeTupleTime(*): 0
            - ScannerThreadsSysTime: 0
            - ScannerThreadsUserTime: 0
          - ScannerThreadsVoluntaryContextSwitches: 0
          - TotalRawHdfsReadTime(*): 1240520546
          - TotalReadThroughput: 25085300
          - TotalTime: 1551352044
      Instance f04dba71f0b5a296:f4f5392c0729d4bb (host=x.xdn3.x.x:22000)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/355.81 MB 3:1/355.76 MB 4:2/711.46 MB 6:1/355.78 MB 7:2/711.67 MB 10:2/711.50 MB 
        - AsyncTotalTime: 0
        - AverageThreadTokens: 10.0
        - InactiveTotalTime: 0
        - PeakMemoryUsage: 1879723448
        - PrepareTime: 108328
        - RowsProduced: 35942400
        - TotalCpuTime: 0
        - TotalNetworkReceiveTime: 0
        - TotalNetworkSendTime: 49449751208
        - TotalStorageWaitTime: 295747308
        - TotalTime: 87837037346
        CodeGen
          - AsyncTotalTime: 0
          - CodegenTime: 0
          - CompileTime: 26177337
          - InactiveTotalTime: 0
          - LoadTime: 31092137
          - ModuleFileSize: 369500
          - TotalTime: 58100661
        DataStreamSender (dst_id=3)
          - AsyncTotalTime: 0
          - BytesSent: 2836403590
          - InactiveTotalTime: 0
          - NetworkThroughput(*): 13001620
          - OverallThroughput: 33206349
          - PeakMemoryUsage: 7264
          - SerializeBatchTime: 35290718042
          - ThriftTransmitTime(*): 218157689310
          - TotalTime: 85417507616
          - UncompressedRowBatchSize: 5865817110
        HDFS_SCAN_NODE (id=0)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/355.81 MB 3:1/355.76 MB 4:2/711.46 MB 6:1/355.78 MB 7:2/711.67 MB 10:2/711.50 MB 
          - AsyncTotalTime: 0
          - AverageHdfsReadThreadConcurrency: 0.022727272727272728
          - AverageScannerThreadConcurrency: 9.0
          - BytesRead: 2314192367
          - BytesReadDataNodeCache: 0
          - BytesReadLocal: 0
          - BytesReadShortCircuit: 0
          - DecompressionTime: 4134891168
          - InactiveTotalTime: 0
          - NumColumns: 0
          - NumDisksAccessed: 0
          - NumScannerThreadsStarted: 9
          - PeakMemoryUsage: 1879568736
          - PerReadThreadRawHdfsThroughput: 1786268967
          - RowsRead: 36105216
          - RowsReturned: 35942400
          - RowsReturnedRate: 15265474
          - ScanRangesComplete: 0
          - ScannerThreadsInvoluntaryContextSwitches: 0
          - ScannerThreadsTotalWallClockTime: 0
            - MaterializeTupleTime(*): 0
            - ScannerThreadsSysTime: 0
            - ScannerThreadsUserTime: 0
          - ScannerThreadsVoluntaryContextSwitches: 0
          - TotalRawHdfsReadTime(*): 1295545301
          - TotalReadThroughput: 26297640
          - TotalTime: 2354489537