Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.12.0
-
ghx-label-4
Description
While loading data into a Kudu table against the latest Impala 2.11.0 I noticed that "Data Stream Manager Deferred RPCs" in the memz isn't accurate.
From memz on worker
Process: Limit=201.73 GB Total=85.41 GB Peak=85.41 GB Buffer Pool: Free Buffers: Total=43.64 MB Buffer Pool: Clean Pages: Total=0 Buffer Pool: Unused Reservation: Total=-17.84 MB Data Stream Service Queue: Limit=10.09 GB Total=0 Peak=512.97 MB Data Stream Manager Deferred RPCs: Total=0 Peak=0 TCMalloc Overhead: Total=124.07 MB Free Disk IO Buffers: Total=984.97 MB Peak=984.97 MB RequestPool=root.default: Total=83.92 GB Peak=83.92 GB Query(844a0200d7876345:20bb38b900000000): Reservation=70.44 GB ReservationLimit=161.39 GB OtherMemory=13.48 GB Total=83.92 GB Peak=83.92 GB Fragment 844a0200d7876345:20bb38b9000000a3: Reservation=70.44 GB OtherMemory=38.08 MB Total=70.47 GB Peak=70.47 GB SORT_NODE (id=2): Reservation=70.44 GB OtherMemory=8.00 KB Total=70.44 GB Peak=70.44 GB EXCHANGE_NODE (id=1): Reservation=18.06 MB OtherMemory=0 Total=18.06 MB Peak=19.53 MB KrpcDeferredRpcs: Total=0 Peak=1.47 MB KuduTableSink: Total=20.00 MB Peak=20.00 MB CodeGen: Total=438.00 B Peak=306.00 KB Fragment 844a0200d7876345:20bb38b900000022: Reservation=0 OtherMemory=13.44 GB Total=13.44 GB Peak=13.97 GB HDFS_SCAN_NODE (id=0): Total=13.44 GB Peak=13.97 GB KrpcDataStreamSender (dst_id=1): Total=2.57 MB Peak=3.61 MB CodeGen: Total=234.00 B Peak=52.50 KB Untracked Memory: Total=389.18 MB
And snapshot from query profile
Instance 844a0200d7876345:20bb38b9000000a3 (host=va1030.halxg.cloudera.com:22000):(Total: 1s172ms, non-child: 200.411ms, % non-child: 17.09%)
Fragment Instance Lifecycle Event Timeline: 1s173ms
- Prepare Finished: 199.691ms (199.691ms)
- Open Finished: 1s173ms (973.902ms)
MemoryUsage(1m4s): 4.77 GB, 13.21 GB, 19.60 GB, 23.70 GB, 26.67 GB, 29.21 GB, 31.50 GB, 33.63 GB, 35.40 GB, 37.14 GB, 38.54 GB, 39.79 GB, 41.09 GB, 42.37 GB, 43.60 GB, 44.80 GB, 45.95 GB, 47.01 GB, 48.09 GB, 49.17 GB, 50.22 GB, 51.21 GB, 52.40 GB, 53.46 GB, 54.58 GB, 55.61 GB, 56.58 GB, 57.53 GB, 58.45 GB, 59.39 GB, 60.31 GB, 61.20 GB, 62.12 GB, 63.04 GB, 64.15 GB, 65.11 GB, 66.15 GB, 67.06 GB, 67.87 GB, 68.66 GB, 69.49 GB, 70.29 GB, 71.07 GB, 71.83 GB, 72.61 GB
ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 73.69 GB (79123530166)
- PeakReservation: 73.65 GB (79083601920)
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 87.02 GB (93435941671)
- RowsProduced: 0 (0)
- TotalNetworkReceiveTime: 38m28s
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 7 (7)
- TotalThreadsTotalWallClockTime: 973.873ms
- TotalThreadsSysTime: 2.000ms
- TotalThreadsUserTime: 55.991ms
- TotalThreadsVoluntaryContextSwitches: 25 (25)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 0.000ns
- ExecTreeExecTime: 0.000ns
- OpenTime: 973.876ms
- ExecTreeOpenTime: 915.567ms
- PrepareTime: 198.988ms
- ExecTreePrepareTime: 155.134us
KuduTableSink:(Total: 12.589us, non-child: 12.589us, % non-child: 100.00%)
- KuduApplyTimer: 0.000ns
- NumRowErrors: 0 (0)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 0
- TotalNumRows: 0 (0)
SORT_NODE (id=2):(Total: 915.718ms, non-child: 0.000ns, % non-child: 0.00%)
SortType: Partial
ExecOption: Codegen Enabled
- NumRowsPerRun: 0 (0) (Number of samples: 0)
- InMemorySortTime: 0.000ns
- PeakMemoryUsage: 73.65 GB (79083610112)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- RunsCreated: 1 (1)
- SortDataSize: 0
Buffer pool:
- AllocTime: 3m9s
- CumulativeAllocationBytes: 73.65 GB (79083601920)
- CumulativeAllocations: 37.71K (37710)
- PeakReservation: 73.65 GB (79083601920)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 73.65 GB (79083601920)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
EXCHANGE_NODE (id=1):(Total: 38m39s, non-child: 18s181ms, % non-child: 0.78%)
- ConvertRowBatchTime: 7s926ms
- PeakMemoryUsage: 19.53 MB (20481319)
- RowsReturned: 288.83M (288830892)
- RowsReturnedRate: 124.54 K/sec
Buffer pool:
- AllocTime: 17m12s
- CumulativeAllocationBytes: 131.17 GB (140839444480)
- CumulativeAllocations: 6.88M (6876926)
- PeakReservation: 18.06 MB (18939904)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 18.06 MB (18939904)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
RecvrSide:
BytesReceived(1m4s): 4.87 GB, 13.37 GB, 19.82 GB, 23.95 GB, 26.95 GB, 29.52 GB, 31.82 GB, 33.98 GB, 35.77 GB, 37.52 GB, 38.94 GB, 40.20 GB, 41.51 GB, 42.80 GB, 44.05 GB, 45.26 GB, 46.42 GB, 47.49 GB, 48.58 GB, 49.68 GB, 50.74 GB, 51.74 GB, 52.94 GB, 54.01 GB, 55.15 GB, 56.19 GB, 57.16 GB, 58.12 GB, 59.05 GB, 60.00 GB, 60.93 GB, 61.84 GB, 62.76 GB, 63.70 GB, 64.82 GB, 65.79 GB, 66.84 GB, 67.76 GB, 68.57 GB, 69.38 GB, 70.22 GB, 71.02 GB, 71.81 GB, 72.58 GB, 73.37 GB
- FirstBatchArrivalWaitTime: 915.459ms
- TotalBytesReceived: 74.45 GB (79944017735)
- TotalGetBatchTime: 38m28s
- DataArrivalTimer: 38m21s
SenderSide:
- DeserializeRowBatchTime: 20m44s
- NumBatchesArrived: 3.44M (3438463)
- NumBatchesDeferred: 27.14K (27144)
- NumBatchesEnqueued: 3.44M (3438463)
- NumBatchesReceived: 3.44M (3438463)
- NumEarlySenders: 0 (0)
- NumEosReceived: 0 (0)