Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7300 Clean up items from KRPC milestone 1
  3. IMPALA-6684

Large payload going through exchange results in lots of untracked memory in the form of a string

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 2.11.0
    • Impala 4.2.0
    • Distributed Exec
    • None
    • ghx-label-4

    Description

      While running a query that shuffles 1GB rows noticed that there is a large amount of untracked memory in the form of a string, this issue repros with/without KRPC.

      tgar identified that the memory is from tuple_data_ and that the while the payload is transient much of the memory untracked. 

      https://github.com/apache/impala/blob/master/be/src/runtime/row-batch.h#L88

      Process: Limit=200.00 GB Total=22.85 GB Peak=201.03 GB
        Buffer Pool: Free Buffers: Total=4.13 GB
        Buffer Pool: Clean Pages: Total=0
        Buffer Pool: Unused Reservation: Total=-2.00 GB
        Data Stream Service Queue: Limit=10.00 GB Total=0 Peak=1.00 GB
        Data Stream Manager Deferred RPCs: Total=0 Peak=27.70 KB
        TCMalloc Overhead: Total=206.90 MB
        Free Disk IO Buffers: Total=1.15 GB Peak=1.89 GB
        RequestPool=root.jenkins: Total=0 Peak=13.32 GB
        RequestPool=root.mmokhtar: Total=13.67 GB Peak=117.15 GB
          Query(ec4cab0940313fdd:d6196e4800000000): Reservation=72.00 MB ReservationLimit=160.00 GB OtherMemory=13.60 GB Total=13.67 GB Peak=13.67 GB
            Unclaimed reservations: Reservation=2.00 MB OtherMemory=0 Total=2.00 MB Peak=176.00 MB
            Fragment ec4cab0940313fdd:d6196e480000002b: Reservation=34.00 MB OtherMemory=4.00 GB Total=4.03 GB Peak=4.03 GB
              AGGREGATION_NODE (id=11): Total=24.00 KB Peak=24.00 KB
                Exprs: Total=4.00 KB Peak=4.00 KB
              HASH_JOIN_NODE (id=10): Reservation=34.00 MB OtherMemory=34.25 KB Total=34.03 MB Peak=34.05 MB
                Exprs: Total=13.12 KB Peak=13.12 KB
                Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=21.12 KB
                  Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB Peak=13.12 KB
              EXCHANGE_NODE (id=20): Reservation=4.00 GB OtherMemory=0 Total=4.00 GB Peak=4.00 GB
                KrpcDeferredRpcs: Total=0 Peak=0
              EXCHANGE_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              KrpcDataStreamSender (dst_id=22): Total=5.12 KB Peak=5.12 KB
              CodeGen: Total=7.79 KB Peak=1.46 MB
            Fragment ec4cab0940313fdd:d6196e480000000f: Reservation=36.00 MB OtherMemory=9.60 GB Total=9.63 GB Peak=9.63 GB
              UNION_NODE (id=0): Total=2.09 GB Peak=2.09 GB
                Exprs: Total=2.07 GB Peak=2.07 GB
              AGGREGATION_NODE (id=13): Reservation=36.00 MB OtherMemory=5.44 GB Total=5.48 GB Peak=5.48 GB
                Exprs: Total=5.44 GB Peak=5.44 GB
              EXCHANGE_NODE (id=12): Reservation=0 OtherMemory=0 Total=0 Peak=57.54 MB
                KrpcDeferredRpcs: Total=0 Peak=8.30 MB
              AGGREGATION_NODE (id=15): Total=17.12 KB Peak=17.12 KB
                Exprs: Total=17.12 KB Peak=17.12 KB
              EXCHANGE_NODE (id=14): Reservation=16.00 KB OtherMemory=0 Total=16.00 KB Peak=16.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              KrpcDataStreamSender (dst_id=20): Total=1.03 GB Peak=1.03 GB
              CodeGen: Total=8.04 KB Peak=1.02 MB
            Fragment ec4cab0940313fdd:d6196e4800000007: Reservation=0 OtherMemory=0 Total=0 Peak=6.36 GB
              AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=5.92 GB
              HDFS_SCAN_NODE (id=1): Total=0 Peak=535.01 MB
              KrpcDataStreamSender (dst_id=12): Total=0 Peak=84.57 MB
              CodeGen: Total=0 Peak=592.00 KB
            Fragment ec4cab0940313fdd:d6196e480000000e: Reservation=0 OtherMemory=0 Total=0 Peak=2.34 MB
              AGGREGATION_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
              HDFS_SCAN_NODE (id=3): Total=0 Peak=280.00 KB
              KrpcDataStreamSender (dst_id=14): Total=0 Peak=35.82 KB
              CodeGen: Total=0 Peak=622.00 KB
            Fragment ec4cab0940313fdd:d6196e4800000024: Reservation=0 OtherMemory=0 Total=0 Peak=34.22 MB
              UNION_NODE (id=5): Total=0 Peak=0
              AGGREGATION_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=34.02 MB
              EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=64.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              AGGREGATION_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=1.96 MB
              EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              KrpcDataStreamSender (dst_id=21): Total=0 Peak=103.53 KB
              CodeGen: Total=0 Peak=491.50 KB
            Fragment ec4cab0940313fdd:d6196e4800000023: Reservation=0 OtherMemory=0 Total=0 Peak=2.67 MB
              AGGREGATION_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
              HDFS_SCAN_NODE (id=8): Total=0 Peak=552.00 KB
              KrpcDataStreamSender (dst_id=18): Total=0 Peak=99.53 KB
              CodeGen: Total=0 Peak=504.50 KB
            Fragment ec4cab0940313fdd:d6196e480000001c: Reservation=0 OtherMemory=0 Total=0 Peak=280.68 MB
              AGGREGATION_NODE (id=7): Reservation=0 OtherMemory=0 Total=0 Peak=34.05 MB
              HDFS_SCAN_NODE (id=6): Total=0 Peak=246.54 MB
              KrpcDataStreamSender (dst_id=16): Total=0 Peak=127.53 KB
              CodeGen: Total=0 Peak=474.50 KB
        Untracked Memory: Total=5.70 GB
      

      Attachments

        1. Untracked memory profile.txt
          425 kB
          Mostafa Mokhtar
        2. heap_profile_growth_40GB.pdf
          13 kB
          Mostafa Mokhtar

        Activity

          People

            oshahidi Omid Shahidi
            mmokhtar Mostafa Mokhtar
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: