[IMPALA-6657] Investigate why memory allocation in Exchange receiver node takes a long time

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Component/s: Backend
• Labels: memorymanager performance scalability
• Epic Color: ghx-label-9
• Assignee: Mostafa Mokhtar
• Reporter: Mostafa Mokhtar
• Created: 14/Mar/18 00:28
• Updated: 28/Jun/18 21:27
• Resolved: 28/Jun/18 21:27

Description

It was observed while inserting large amounts of data into a Kudu table Exchange operator was running slow, query profile showed a big portion of the time was spent in memory allocation in the buffer pool

      EXCHANGE_NODE (id=1):(Total: 5h53m, non-child: 48s853ms, % non-child: 0.23%)
           - ConvertRowBatchTime: 20s289ms
           - PeakMemoryUsage: 19.53 MB (20483562)
           - RowsReturned: 575.30M (575298780)
           - RowsReturnedRate: 27.10 K/sec
          Buffer pool:
             - AllocTime: 2h53m
             - CumulativeAllocationBytes: 261.26 GB (280526643200)
             - CumulativeAllocations: 13.70M (13697590)
             - PeakReservation: 18.06 MB (18939904)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 18.06 MB (18939904)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          RecvrSide:
            BytesReceived(8m32s): 20.91 GB, 37.03 GB, 45.62 GB, 53.22 GB, 60.17 GB, 66.30 GB, 71.60 GB, 76.59 GB, 81.36 GB, 86.03 GB, 90.35 GB, 94.30 GB, 98.17 GB, 101.98 GB, 105.58 GB, 109.08 GB, 112.33 GB, 115.47 GB, 118.45 GB, 121.30 GB, 124.09 GB, 126.74 GB, 129.26 GB, 131.88 GB, 134.41 GB, 136.85 GB, 139.32 GB, 141.77 GB, 144.23 GB, 146.71 GB, 148.26 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.29 GB, 148.30 GB, 148.30 GB, 148.30 GB, 148.30 GB, 148.30 GB, 148.30 GB, 148.30 GB, 148.30 GB
             - FirstBatchArrivalWaitTime: 1s071ms
             - TotalBytesReceived: 148.30 GB (159234237617)
             - TotalGetBatchTime: 5h53m
               - DataArrivalTimer: 5h52m
          SenderSide:
             - DeserializeRowBatchTime: 3h4m
             - NumBatchesArrived: 6.85M (6848795)
             - NumBatchesDeferred: 99.67K (99667)
             - NumBatchesEnqueued: 6.85M (6848795)
             - NumBatchesReceived: 6.85M (6848795)
             - NumEarlySenders: 0 (0)
             - NumEosReceived: 0 (0)