[IMPALA-6684] Large payload going through exchange results in lots of untracked memory in the form of a string

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0
• Fix Version/s: Impala 4.2.0
• Component/s: Distributed Exec
• Epic Color: ghx-label-4
• Assignee: Omid Shahidi
• Reporter: Mostafa Mokhtar
• Created: 16/Mar/18 04:57
• Updated: 13/Sep/24 05:42
• Resolved: 19/Oct/22 02:07

Description

While running a query that shuffles 1GB rows noticed that there is a large amount of untracked memory in the form of a string, this issue repros with/without KRPC.

tgar identified that the memory is from tuple_data_ and that the while the payload is transient much of the memory untracked. 

https://github.com/apache/impala/blob/master/be/src/runtime/row-batch.h#L88

Process: Limit=200.00 GB Total=22.85 GB Peak=201.03 GB
  Buffer Pool: Free Buffers: Total=4.13 GB
  Buffer Pool: Clean Pages: Total=0
  Buffer Pool: Unused Reservation: Total=-2.00 GB
  Data Stream Service Queue: Limit=10.00 GB Total=0 Peak=1.00 GB
  Data Stream Manager Deferred RPCs: Total=0 Peak=27.70 KB
  TCMalloc Overhead: Total=206.90 MB
  Free Disk IO Buffers: Total=1.15 GB Peak=1.89 GB
  RequestPool=root.jenkins: Total=0 Peak=13.32 GB
  RequestPool=root.mmokhtar: Total=13.67 GB Peak=117.15 GB
    Query(ec4cab0940313fdd:d6196e4800000000): Reservation=72.00 MB ReservationLimit=160.00 GB OtherMemory=13.60 GB Total=13.67 GB Peak=13.67 GB
      Unclaimed reservations: Reservation=2.00 MB OtherMemory=0 Total=2.00 MB Peak=176.00 MB
      Fragment ec4cab0940313fdd:d6196e480000002b: Reservation=34.00 MB OtherMemory=4.00 GB Total=4.03 GB Peak=4.03 GB
        AGGREGATION_NODE (id=11): Total=24.00 KB Peak=24.00 KB
          Exprs: Total=4.00 KB Peak=4.00 KB
        HASH_JOIN_NODE (id=10): Reservation=34.00 MB OtherMemory=34.25 KB Total=34.03 MB Peak=34.05 MB
          Exprs: Total=13.12 KB Peak=13.12 KB
          Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=21.12 KB
            Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB Peak=13.12 KB
        EXCHANGE_NODE (id=20): Reservation=4.00 GB OtherMemory=0 Total=4.00 GB Peak=4.00 GB
          KrpcDeferredRpcs: Total=0 Peak=0
        EXCHANGE_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        KrpcDataStreamSender (dst_id=22): Total=5.12 KB Peak=5.12 KB
        CodeGen: Total=7.79 KB Peak=1.46 MB
      Fragment ec4cab0940313fdd:d6196e480000000f: Reservation=36.00 MB OtherMemory=9.60 GB Total=9.63 GB Peak=9.63 GB
        UNION_NODE (id=0): Total=2.09 GB Peak=2.09 GB
          Exprs: Total=2.07 GB Peak=2.07 GB
        AGGREGATION_NODE (id=13): Reservation=36.00 MB OtherMemory=5.44 GB Total=5.48 GB Peak=5.48 GB
          Exprs: Total=5.44 GB Peak=5.44 GB
        EXCHANGE_NODE (id=12): Reservation=0 OtherMemory=0 Total=0 Peak=57.54 MB
          KrpcDeferredRpcs: Total=0 Peak=8.30 MB
        AGGREGATION_NODE (id=15): Total=17.12 KB Peak=17.12 KB
          Exprs: Total=17.12 KB Peak=17.12 KB
        EXCHANGE_NODE (id=14): Reservation=16.00 KB OtherMemory=0 Total=16.00 KB Peak=16.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        KrpcDataStreamSender (dst_id=20): Total=1.03 GB Peak=1.03 GB
        CodeGen: Total=8.04 KB Peak=1.02 MB
      Fragment ec4cab0940313fdd:d6196e4800000007: Reservation=0 OtherMemory=0 Total=0 Peak=6.36 GB
        AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=5.92 GB
        HDFS_SCAN_NODE (id=1): Total=0 Peak=535.01 MB
        KrpcDataStreamSender (dst_id=12): Total=0 Peak=84.57 MB
        CodeGen: Total=0 Peak=592.00 KB
      Fragment ec4cab0940313fdd:d6196e480000000e: Reservation=0 OtherMemory=0 Total=0 Peak=2.34 MB
        AGGREGATION_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
        HDFS_SCAN_NODE (id=3): Total=0 Peak=280.00 KB
        KrpcDataStreamSender (dst_id=14): Total=0 Peak=35.82 KB
        CodeGen: Total=0 Peak=622.00 KB
      Fragment ec4cab0940313fdd:d6196e4800000024: Reservation=0 OtherMemory=0 Total=0 Peak=34.22 MB
        UNION_NODE (id=5): Total=0 Peak=0
        AGGREGATION_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=34.02 MB
        EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=64.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        AGGREGATION_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=1.96 MB
        EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        KrpcDataStreamSender (dst_id=21): Total=0 Peak=103.53 KB
        CodeGen: Total=0 Peak=491.50 KB
      Fragment ec4cab0940313fdd:d6196e4800000023: Reservation=0 OtherMemory=0 Total=0 Peak=2.67 MB
        AGGREGATION_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
        HDFS_SCAN_NODE (id=8): Total=0 Peak=552.00 KB
        KrpcDataStreamSender (dst_id=18): Total=0 Peak=99.53 KB
        CodeGen: Total=0 Peak=504.50 KB
      Fragment ec4cab0940313fdd:d6196e480000001c: Reservation=0 OtherMemory=0 Total=0 Peak=280.68 MB
        AGGREGATION_NODE (id=7): Reservation=0 OtherMemory=0 Total=0 Peak=34.05 MB
        HDFS_SCAN_NODE (id=6): Total=0 Peak=246.54 MB
        KrpcDataStreamSender (dst_id=16): Total=0 Peak=127.53 KB
        CodeGen: Total=0 Peak=474.50 KB
  Untracked Memory: Total=5.70 GB

Issue Links

• relates to: IMPALA-12594 KrpcDataStreamSender's mem estimate is different than real usage