Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6652

KRPC : Data Stream Manager Deferred RPCs in memz page should be renamed

    Details

    • Epic Color:
      ghx-label-4

      Description

      While loading data into a Kudu table against the latest Impala 2.11.0 I noticed that "Data Stream Manager Deferred RPCs" in the memz isn't accurate.
       
      From memz on worker

      Process: Limit=201.73 GB Total=85.41 GB Peak=85.41 GB
        Buffer Pool: Free Buffers: Total=43.64 MB
        Buffer Pool: Clean Pages: Total=0
        Buffer Pool: Unused Reservation: Total=-17.84 MB
        Data Stream Service Queue: Limit=10.09 GB Total=0 Peak=512.97 MB
        Data Stream Manager Deferred RPCs: Total=0 Peak=0
        TCMalloc Overhead: Total=124.07 MB
        Free Disk IO Buffers: Total=984.97 MB Peak=984.97 MB
        RequestPool=root.default: Total=83.92 GB Peak=83.92 GB
          Query(844a0200d7876345:20bb38b900000000): Reservation=70.44 GB ReservationLimit=161.39 GB OtherMemory=13.48 GB Total=83.92 GB Peak=83.92 GB
            Fragment 844a0200d7876345:20bb38b9000000a3: Reservation=70.44 GB OtherMemory=38.08 MB Total=70.47 GB Peak=70.47 GB
              SORT_NODE (id=2): Reservation=70.44 GB OtherMemory=8.00 KB Total=70.44 GB Peak=70.44 GB
              EXCHANGE_NODE (id=1): Reservation=18.06 MB OtherMemory=0 Total=18.06 MB Peak=19.53 MB
                KrpcDeferredRpcs: Total=0 Peak=1.47 MB
              KuduTableSink: Total=20.00 MB Peak=20.00 MB
              CodeGen: Total=438.00 B Peak=306.00 KB
            Fragment 844a0200d7876345:20bb38b900000022: Reservation=0 OtherMemory=13.44 GB Total=13.44 GB Peak=13.97 GB
              HDFS_SCAN_NODE (id=0): Total=13.44 GB Peak=13.97 GB
              KrpcDataStreamSender (dst_id=1): Total=2.57 MB Peak=3.61 MB
              CodeGen: Total=234.00 B Peak=52.50 KB
        Untracked Memory: Total=389.18 MB
      

       
      And snapshot from query profile

       Instance 844a0200d7876345:20bb38b9000000a3 (host=va1030.halxg.cloudera.com:22000):(Total: 1s172ms, non-child: 200.411ms, % non-child: 17.09%)
              Fragment Instance Lifecycle Event Timeline: 1s173ms
                 - Prepare Finished: 199.691ms (199.691ms)
                 - Open Finished: 1s173ms (973.902ms)
              MemoryUsage(1m4s): 4.77 GB, 13.21 GB, 19.60 GB, 23.70 GB, 26.67 GB, 29.21 GB, 31.50 GB, 33.63 GB, 35.40 GB, 37.14 GB, 38.54 GB, 39.79 GB, 41.09 GB, 42.37 GB, 43.60 GB, 44.80 GB, 45.95 GB, 47.01 GB, 48.09 GB, 49.17 GB, 50.22 GB, 51.21 GB, 52.40 GB, 53.46 GB, 54.58 GB, 55.61 GB, 56.58 GB, 57.53 GB, 58.45 GB, 59.39 GB, 60.31 GB, 61.20 GB, 62.12 GB, 63.04 GB, 64.15 GB, 65.11 GB, 66.15 GB, 67.06 GB, 67.87 GB, 68.66 GB, 69.49 GB, 70.29 GB, 71.07 GB, 71.83 GB, 72.61 GB
              ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
               - AverageThreadTokens: 1.00 
               - BloomFilterBytes: 0
               - PeakMemoryUsage: 73.69 GB (79123530166)
               - PeakReservation: 73.65 GB (79083601920)
               - PeakUsedReservation: 0
               - PerHostPeakMemUsage: 87.02 GB (93435941671)
               - RowsProduced: 0 (0)
               - TotalNetworkReceiveTime: 38m28s
               - TotalNetworkSendTime: 0.000ns
               - TotalStorageWaitTime: 0.000ns
               - TotalThreadsInvoluntaryContextSwitches: 7 (7)
               - TotalThreadsTotalWallClockTime: 973.873ms
                 - TotalThreadsSysTime: 2.000ms
                 - TotalThreadsUserTime: 55.991ms
               - TotalThreadsVoluntaryContextSwitches: 25 (25)
              Buffer pool:
                 - AllocTime: 0.000ns
                 - CumulativeAllocationBytes: 0
                 - CumulativeAllocations: 0 (0)
                 - PeakReservation: 0
                 - PeakUnpinnedBytes: 0
                 - PeakUsedReservation: 0
                 - ReadIoBytes: 0
                 - ReadIoOps: 0 (0)
                 - ReadIoWaitTime: 0.000ns
                 - ReservationLimit: 0
                 - WriteIoBytes: 0
                 - WriteIoOps: 0 (0)
                 - WriteIoWaitTime: 0.000ns
              Fragment Instance Lifecycle Timings:
                 - ExecTime: 0.000ns
                   - ExecTreeExecTime: 0.000ns
                 - OpenTime: 973.876ms
                   - ExecTreeOpenTime: 915.567ms
                 - PrepareTime: 198.988ms
                   - ExecTreePrepareTime: 155.134us
              KuduTableSink:(Total: 12.589us, non-child: 12.589us, % non-child: 100.00%)
                 - KuduApplyTimer: 0.000ns
                 - NumRowErrors: 0 (0)
                 - PeakMemoryUsage: 20.00 MB (20971520)
                 - RowsProcessedRate: 0
                 - TotalNumRows: 0 (0)
              SORT_NODE (id=2):(Total: 915.718ms, non-child: 0.000ns, % non-child: 0.00%)
                SortType: Partial
                ExecOption: Codegen Enabled
                - NumRowsPerRun: 0 (0) (Number of samples: 0)
                 - InMemorySortTime: 0.000ns
                 - PeakMemoryUsage: 73.65 GB (79083610112)
                 - RowsReturned: 0 (0)
                 - RowsReturnedRate: 0
                 - RunsCreated: 1 (1)
                 - SortDataSize: 0
                Buffer pool:
                   - AllocTime: 3m9s
                   - CumulativeAllocationBytes: 73.65 GB (79083601920)
                   - CumulativeAllocations: 37.71K (37710)
                   - PeakReservation: 73.65 GB (79083601920)
                   - PeakUnpinnedBytes: 0
                   - PeakUsedReservation: 73.65 GB (79083601920)
                   - ReadIoBytes: 0
                   - ReadIoOps: 0 (0)
                   - ReadIoWaitTime: 0.000ns
                   - WriteIoBytes: 0
                   - WriteIoOps: 0 (0)
                   - WriteIoWaitTime: 0.000ns
              EXCHANGE_NODE (id=1):(Total: 38m39s, non-child: 18s181ms, % non-child: 0.78%)
                 - ConvertRowBatchTime: 7s926ms
                 - PeakMemoryUsage: 19.53 MB (20481319)
                 - RowsReturned: 288.83M (288830892)
                 - RowsReturnedRate: 124.54 K/sec
                Buffer pool:
                   - AllocTime: 17m12s
                   - CumulativeAllocationBytes: 131.17 GB (140839444480)
                   - CumulativeAllocations: 6.88M (6876926)
                   - PeakReservation: 18.06 MB (18939904)
                   - PeakUnpinnedBytes: 0
                   - PeakUsedReservation: 18.06 MB (18939904)
                   - ReadIoBytes: 0
                   - ReadIoOps: 0 (0)
                   - ReadIoWaitTime: 0.000ns
                   - WriteIoBytes: 0
                   - WriteIoOps: 0 (0)
                   - WriteIoWaitTime: 0.000ns
                RecvrSide:
                  BytesReceived(1m4s): 4.87 GB, 13.37 GB, 19.82 GB, 23.95 GB, 26.95 GB, 29.52 GB, 31.82 GB, 33.98 GB, 35.77 GB, 37.52 GB, 38.94 GB, 40.20 GB, 41.51 GB, 42.80 GB, 44.05 GB, 45.26 GB, 46.42 GB, 47.49 GB, 48.58 GB, 49.68 GB, 50.74 GB, 51.74 GB, 52.94 GB, 54.01 GB, 55.15 GB, 56.19 GB, 57.16 GB, 58.12 GB, 59.05 GB, 60.00 GB, 60.93 GB, 61.84 GB, 62.76 GB, 63.70 GB, 64.82 GB, 65.79 GB, 66.84 GB, 67.76 GB, 68.57 GB, 69.38 GB, 70.22 GB, 71.02 GB, 71.81 GB, 72.58 GB, 73.37 GB
                   - FirstBatchArrivalWaitTime: 915.459ms
                   - TotalBytesReceived: 74.45 GB (79944017735)
                   - TotalGetBatchTime: 38m28s
                     - DataArrivalTimer: 38m21s
                SenderSide:
                   - DeserializeRowBatchTime: 20m44s
                   - NumBatchesArrived: 3.44M (3438463)
                   - NumBatchesDeferred: 27.14K (27144)
                   - NumBatchesEnqueued: 3.44M (3438463)
                   - NumBatchesReceived: 3.44M (3438463)
                   - NumEarlySenders: 0 (0)
                   - NumEosReceived: 0 (0)
        

        Attachments

          Activity

            People

            • Assignee:
              lv Lars Volker
              Reporter:
              mmokhtar Mostafa Mokhtar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: