Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6792

Appears to be a memory leak in orphaned fragments

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
    • Fix Version/s: Impala 2.12.0
    • Component/s: Backend
    • Labels:
      None

      Description

      On Impala daemons that have orphaned fragments it appears that memory continued to grow after query got cancelled, indicating of a memory leak.
      Memz page indicate that some fragments continue to run.

      Memory Usage
      Memory consumption / limit: 101.48 GB / 64.00 GB
      Breakdown
      Process: memory limit exceeded. Limit=64.00 GB Total=101.51 GB Peak=101.51 GB
        Buffer Pool: Free Buffers: Total=0
        Buffer Pool: Clean Pages: Total=0
        Buffer Pool: Unused Reservation: Total=-71.09 MB
        Data Stream Service Queue: Limit=3.20 GB Total=0 Peak=875.26 KB
        Data Stream Manager Early RPCs: Total=0 Peak=23.03 MB
        TCMalloc Overhead: Total=760.53 MB
        Free Disk IO Buffers: Total=1.24 GB Peak=1.30 GB
        RequestPool=root.default: Total=909.59 MB Peak=11.91 GB
          Query(6c4aa5355486f3f6:5125570400000000): Limit=6.00 GB Reservation=69.50 MB ReservationLimit=4.80 GB OtherMemory=67.90 MB Total=137.40 MB Peak=400.25 MB
            Unclaimed reservations: Reservation=34.00 MB OtherMemory=0 Total=34.00 MB Peak=90.14 MB
            Fragment 6c4aa5355486f3f6:5125570400000845: Reservation=35.50 MB OtherMemory=31.87 MB Total=67.37 MB Peak=67.37 MB
              Runtime Filter Bank: Reservation=512.00 KB ReservationLimit=512.00 KB OtherMemory=0 Total=512.00 KB Peak=512.00 KB
              AGGREGATION_NODE (id=12): Total=4.00 KB Peak=4.00 KB
                Exprs: Total=4.00 KB Peak=4.00 KB
              AGGREGATION_NODE (id=11): Total=17.12 KB Peak=17.12 KB
                Exprs: Total=17.12 KB Peak=17.12 KB
              HASH_JOIN_NODE (id=10): Reservation=17.00 MB OtherMemory=66.25 KB Total=17.06 MB Peak=17.06 MB
                Exprs: Total=13.12 KB Peak=13.12 KB
                Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=13.12 KB
                  Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB Peak=13.12 KB
              EXCHANGE_NODE (id=19): Reservation=10.59 MB OtherMemory=6.69 MB Total=17.29 MB Peak=17.29 MB
                KrpcDeferredRpcs: Total=6.69 MB Peak=6.69 MB
              HASH_JOIN_NODE (id=9): Reservation=18.00 MB OtherMemory=34.25 KB Total=18.03 MB Peak=18.10 MB
                Exprs: Total=13.12 KB Peak=13.12 KB
                Hash Join Builder (join_node_id=9): Total=13.12 KB Peak=45.12 KB
                  Hash Join Builder (join_node_id=9) Exprs: Total=13.12 KB Peak=13.12 KB
              EXCHANGE_NODE (id=17): Reservation=14.44 MB OtherMemory=0 Total=14.44 MB Peak=14.45 MB
                KrpcDeferredRpcs: Total=0 Peak=2.42 MB
              EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=776.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              KrpcDataStreamSender (dst_id=20): Total=3.12 KB Peak=3.12 KB
              CodeGen: Total=18.24 KB Peak=2.89 MB
            Fragment 6c4aa5355486f3f6:5125570400000047: Reservation=0 OtherMemory=0 Total=0 Peak=41.25 MB
              HDFS_SCAN_NODE (id=4): Total=0 Peak=18.75 MB
              KrpcDataStreamSender (dst_id=17): Total=0 Peak=22.19 MB
              CodeGen: Total=0 Peak=222.50 KB
            Fragment 6c4aa5355486f3f6:512557040000033f: Reservation=0 OtherMemory=0 Total=0 Peak=241.80 MB
              HASH_JOIN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=2.00 MB
                Hash Join Builder (join_node_id=8): Total=0 Peak=21.12 KB
              HASH_JOIN_NODE (id=7): Reservation=0 OtherMemory=0 Total=0 Peak=1.99 MB
                Hash Join Builder (join_node_id=7): Total=0 Peak=21.12 KB
              HASH_JOIN_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=128.08 MB
                Hash Join Builder (join_node_id=6): Total=0 Peak=21.12 KB
              HDFS_SCAN_NODE (id=0): Total=0 Peak=106.87 MB
              EXCHANGE_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=2.46 MB
                KrpcDeferredRpcs: Total=0 Peak=388.27 KB
              EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              KrpcDataStreamSender (dst_id=18): Total=0 Peak=2.61 MB
              CodeGen: Total=0 Peak=3.61 MB
            Fragment 6c4aa5355486f3f6:51255704000002e8: Reservation=0 OtherMemory=0 Total=0 Peak=6.71 MB
              HDFS_SCAN_NODE (id=2): Total=0 Peak=3.44 MB
              KrpcDataStreamSender (dst_id=14): Total=0 Peak=3.23 MB
              CodeGen: Total=0 Peak=241.50 KB
            Fragment 6c4aa5355486f3f6:51255704000005ea: Reservation=0 OtherMemory=36.02 MB Total=36.02 MB Peak=46.83 MB
              HDFS_SCAN_NODE (id=5): Total=9.55 MB Peak=18.17 MB
              KrpcDataStreamSender (dst_id=19): Total=26.46 MB Peak=28.87 MB
              CodeGen: Total=234.00 B Peak=52.50 KB
          Query(b44c5962ad16140:4bd217e100000000): Limit=6.00 GB Reservation=660.28 MB ReservationLimit=4.80 GB OtherMemory=111.92 MB Total=772.20 MB Peak=778.99 MB
            Unclaimed reservations: Reservation=3.94 MB OtherMemory=0 Total=3.94 MB Peak=120.66 MB
            Fragment b44c5962ad16140:4bd217e100000bbb: Reservation=0 OtherMemory=49.74 KB Total=49.74 KB Peak=1.41 MB
              SORT_NODE (id=16): Total=8.00 KB Peak=8.00 KB
                Exprs: Total=4.00 KB Peak=4.00 KB
              AGGREGATION_NODE (id=27): Total=25.12 KB Peak=25.12 KB
                Exprs: Total=25.12 KB Peak=25.12 KB
              EXCHANGE_NODE (id=26): Reservation=0 OtherMemory=0 Total=0 Peak=0
                KrpcDeferredRpcs: Total=0 Peak=0
              KrpcDataStreamSender (dst_id=28): Total=3.88 KB Peak=3.88 KB
              CodeGen: Total=4.74 KB Peak=1.37 MB
            Fragment b44c5962ad16140:4bd217e100000934: Reservation=332.39 MB OtherMemory=15.81 MB Total=348.20 MB Peak=358.59 MB
              Runtime Filter Bank: Reservation=2.52 MB ReservationLimit=2.52 MB OtherMemory=0 Total=2.52 MB Peak=2.52 MB
              AGGREGATION_NODE (id=15): Total=25.12 KB Peak=25.12 KB
                Exprs: Total=25.12 KB Peak=25.12 KB
              HASH_JOIN_NODE (id=14): Reservation=1.94 MB OtherMemory=82.25 KB Total=2.02 MB Peak=2.03 MB
                Exprs: Total=13.12 KB Peak=13.12 KB
                Hash Join Builder (join_node_id=14): Total=13.12 KB Peak=21.12 KB
                  Hash Join Builder (join_node_id=14) Exprs: Total=13.12 KB Peak=13.12 KB
              HASH_JOIN_NODE (id=13): Reservation=1.94 MB OtherMemory=74.25 KB Total=2.01 MB Peak=2.03 MB
                Exprs: Total=13.12 KB Peak=13.12 KB
                Hash Join Builder (join_node_id=13): Total=13.12 KB Peak=21.12 KB
                  Hash Join Builder (join_node_id=13) Exprs: Total=13.12 KB Peak=13.12 KB
              HASH_JOIN_NODE (id=12): Reservation=160.00 MB OtherMemory=134.25 KB Total=160.13 MB Peak=160.15 MB
                Exprs: Total=47.12 KB Peak=47.12 KB
                Hash Join Builder (join_node_id=12): Total=47.12 KB Peak=55.12 KB
                  Hash Join Builder (join_node_id=12) Exprs: Total=47.12 KB Peak=47.12 KB
              HASH_JOIN_NODE (id=11): Reservation=160.00 MB OtherMemory=62.25 KB Total=160.06 MB Peak=160.08 MB
                Exprs: Total=17.12 KB Peak=17.12 KB
                Hash Join Builder (join_node_id=11): Total=13.12 KB Peak=21.12 KB
                  Hash Join Builder (join_node_id=11) Exprs: Total=13.12 KB Peak=13.12 KB
              HASH_JOIN_NODE (id=10): Reservation=6.00 MB OtherMemory=54.25 KB Total=6.05 MB Peak=6.07 MB
                Exprs: Total=17.12 KB Peak=17.12 KB
                Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=21.12 KB
                  Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB Peak=13.12 KB
              EXCHANGE_NODE (id=20): Reservation=14.34 MB OtherMemory=0 Total=14.34 MB Peak=14.34 MB
                KrpcDeferredRpcs: Total=0 Peak=1.16 MB
              EXCHANGE_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=2.11 MB
                KrpcDeferredRpcs: Total=0 Peak=232.95 KB
              EXCHANGE_NODE (id=22): Reservation=0 OtherMemory=0 Total=0 Peak=10.97 MB
                KrpcDeferredRpcs: Total=0 Peak=805.37 KB
              EXCHANGE_NODE (id=23): Reservation=0 OtherMemory=0 Total=0 Peak=13.84 MB
                KrpcDeferredRpcs: Total=0 Peak=1.69 MB
              EXCHANGE_NODE (id=24): Reservation=0 OtherMemory=0 Total=0 Peak=40.00 KB
                KrpcDeferredRpcs: Total=0 Peak=8.10 KB
              EXCHANGE_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              KrpcDataStreamSender (dst_id=26): Total=1021.05 KB Peak=1021.05 KB
              CodeGen: Total=48.00 KB Peak=8.65 MB
            Fragment b44c5962ad16140:4bd217e10000054e: Reservation=321.00 MB OtherMemory=20.82 MB Total=341.82 MB Peak=352.83 MB
              Runtime Filter Bank: Reservation=1.00 MB ReservationLimit=1.00 MB OtherMemory=0 Total=1.00 MB Peak=1.00 MB
              HASH_JOIN_NODE (id=9): Reservation=320.00 MB OtherMemory=60.25 KB Total=320.06 MB Peak=320.07 MB
                Exprs: Total=22.12 KB Peak=22.12 KB
                Hash Join Builder (join_node_id=9): Total=22.12 KB Peak=30.12 KB
                  Hash Join Builder (join_node_id=9) Exprs: Total=22.12 KB Peak=22.12 KB
              EXCHANGE_NODE (id=18): Reservation=12.48 MB OtherMemory=0 Total=12.48 MB Peak=16.40 MB
                KrpcDeferredRpcs: Total=0 Peak=4.41 MB
              EXCHANGE_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=4.35 MB
                KrpcDeferredRpcs: Total=0 Peak=1.69 MB
              KrpcDataStreamSender (dst_id=20): Total=8.26 MB Peak=19.26 MB
              CodeGen: Total=8.02 KB Peak=1.34 MB
            Fragment b44c5962ad16140:4bd217e100000048: Reservation=2.95 MB OtherMemory=75.24 MB Total=78.19 MB Peak=86.44 MB
              Runtime Filter Bank: Reservation=1.02 MB ReservationLimit=1.02 MB OtherMemory=0 Total=1.02 MB Peak=1.02 MB
              HASH_JOIN_NODE (id=8): Reservation=1.94 MB OtherMemory=79.25 KB Total=2.01 MB Peak=2.01 MB
                Exprs: Total=13.12 KB Peak=13.12 KB
                Hash Join Builder (join_node_id=8): Total=13.12 KB Peak=21.12 KB
                  Hash Join Builder (join_node_id=8) Exprs: Total=13.12 KB Peak=13.12 KB
              HDFS_SCAN_NODE (id=0): Total=63.00 MB Peak=63.01 MB
              EXCHANGE_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
                KrpcDeferredRpcs: Total=0 Peak=0
              KrpcDataStreamSender (dst_id=18): Total=12.09 MB Peak=20.43 MB
              CodeGen: Total=8.66 KB Peak=1.50 MB
            Fragment b44c5962ad16140:4bd217e1000002f3: Reservation=0 OtherMemory=0 Total=0 Peak=81.41 MB
              HDFS_SCAN_NODE (id=1): Total=0 Peak=60.83 MB
              KrpcDataStreamSender (dst_id=19): Total=0 Peak=19.08 MB
              CodeGen: Total=0 Peak=335.00 KB
            Fragment b44c5962ad16140:4bd217e1000007ef: Reservation=0 OtherMemory=0 Total=0 Peak=26.00 MB
              HDFS_SCAN_NODE (id=5): Total=0 Peak=6.29 MB
              KrpcDataStreamSender (dst_id=21): Total=0 Peak=19.66 MB
              CodeGen: Total=0 Peak=241.50 KB
        Untracked Memory: Total=98.71 GB
      tcmalloc
      ------------------------------------------------
      MALLOC:   108196856848 (103184.6 MiB) Bytes in use by application
      MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
      MALLOC: +    625752432 (  596.8 MiB) Bytes in central cache freelist
      MALLOC: +     25029120 (   23.9 MiB) Bytes in transfer cache freelist
      MALLOC: +    146667648 (  139.9 MiB) Bytes in thread cache freelists
      MALLOC: +    449097920 (  428.3 MiB) Bytes in malloc metadata
      MALLOC:   ------------
      MALLOC: = 109443403968 (104373.4 MiB) Actual memory used (physical + swap)
      MALLOC: +       221184 (    0.2 MiB) Bytes released to OS (aka unmapped)
      MALLOC:   ------------
      MALLOC: = 109443625152 (104373.6 MiB) Virtual address space used
      MALLOC:
      MALLOC:        6575796              Spans in use
      MALLOC:            380              Thread heaps in use
      MALLOC:           8192              Tcmalloc page size
      ------------------------------------------------
      Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
      Bytes released to the OS take up virtual address space but no physical memory.
      System
      Physical Memory: 125.52 GB
      Transparent Huge Pages Config:
        enabled: always [madvise] never
        defrag: always [madvise] never
        khugepaged defrag: 1
      Process and system memory metrics
      Name	Value	Description
      memory.anon-huge-page-bytes	640.00 MB	Total bytes of anonymous (a.k.a. transparent) huge pages used by this process.
      memory.mapped-bytes	137.22 GB	Total bytes of memory mappings in this process (the virtual memory size).
      memory.num-maps	4236	Total number of memory mappings in this process.
      memory.rss	68.40 GB	Resident set size (RSS) of this process, including TCMalloc, buffer pool and Jvm.
      memory.thp.defrag	always [madvise] never	The system-wide 'defrag' setting for Transparent Huge Pages.
      memory.thp.enabled	always [madvise] never	The system-wide 'enabled' setting for Transparent Huge Pages.
      memory.thp.khugepaged-defrag	1	The system-wide 'defrag' setting for khugepaged.
      memory.total-used	101.51 GB	Total memory currently used by TCMalloc and buffer pool.
      Buffer pool memory metrics
      Name	Value	Description
      buffer-pool.clean-page-bytes	0	Total bytes of clean page memory cached in the buffer pool.
      buffer-pool.clean-pages	0	Total number of clean pages cached in the buffer pool.
      buffer-pool.clean-pages-limit	5.44 GB	Limit on number of clean pages cached in the buffer pool.
      buffer-pool.free-buffer-bytes	0	Total bytes of free buffer memory cached in the buffer pool.
      buffer-pool.free-buffers	0	Total number of free buffers cached in the buffer pool.
      buffer-pool.limit	54.40 GB	Maximum allowed bytes allocated by the buffer pool.
      buffer-pool.reserved	781.62 MB	Total bytes of buffers reserved by Impala subsystems
      buffer-pool.system-allocated	710.53 MB	Total buffer memory currently allocated by the buffer pool.
      buffer-pool.unused-reservation-bytes	71.09 MB	Total bytes of buffer reservations by Impala subsystems that are currently unused
      JVM aggregate memory metrics
      Name	Value	Description
      jvm.total.committed-usage-bytes	1.73 GB	Jvm total Committed Usage Bytes
      jvm.total.current-usage-bytes	385.19 MB	Jvm total Current Usage Bytes
      jvm.total.init-usage-bytes	1.88 GB	Jvm total Init Usage Bytes
      jvm.total.max-usage-bytes	31.23 GB	Jvm total Max Usage Bytes
      jvm.total.peak-committed-usage-bytes	2.39 GB	Jvm total Peak Committed Usage Bytes
      jvm.total.peak-current-usage-bytes	1.08 GB	Jvm total Peak Current Usage Bytes
      jvm.total.peak-init-usage-bytes	1.88 GB	Jvm total Peak Init Usage Bytes
      jvm.total.peak-max-usage-bytes	31.32 GB	Jvm total Peak Max Usage Bytes
      JVM heap memory metrics
      Name	Value	Description
      jvm.heap.committed-usage-bytes	1.67 GB	Jvm heap Committed Usage Bytes
      jvm.heap.current-usage-bytes	322.88 MB	Jvm heap Current Usage Bytes
      jvm.heap.init-usage-bytes	1.96 GB	Jvm heap Init Usage Bytes
      jvm.heap.max-usage-bytes	26.67 GB	Jvm heap Max Usage Bytes
      jvm.heap.peak-committed-usage-bytes	0	Jvm heap Peak Committed Usage Bytes
      jvm.heap.peak-current-usage-bytes	0	Jvm heap Peak Current Usage Bytes
      jvm.heap.peak-init-usage-bytes	0	Jvm heap Peak Init Usage Bytes
      jvm.heap.peak-max-usage-bytes	0	Jvm heap Peak Max Usage Bytes
      JVM non-heap memory metrics
      Name	Value	Description
      jvm.non-heap.committed-usage-bytes	63.40 MB	Jvm non-heap Committed Usage Bytes
      jvm.non-heap.current-usage-bytes	62.31 MB	Jvm non-heap Current Usage Bytes
      jvm.non-heap.init-usage-bytes	2.44 MB	Jvm non-heap Init Usage Bytes
      jvm.non-heap.max-usage-bytes	-1.00 B	Jvm non-heap Max Usage Bytes
      jvm.non-heap.peak-committed-usage-bytes	0	Jvm non-heap Peak Committed Usage Bytes
      jvm.non-heap.peak-current-usage-bytes	0	Jvm non-heap Peak Current Usage Bytes
      jvm.non-heap.peak-init-usage-bytes	0	Jvm non-heap Peak Init Usage Bytes
      jvm.non-heap.peak-max-usage-bytes	0	Jvm non-heap Peak Max Usage Bytes
      

        Attachments

        1. memory_leak.txt
          443 kB
          Mostafa Mokhtar
        2. Screen Shot 2018-04-02 at 12.08.54 PM.png
          141 kB
          Mostafa Mokhtar
        3. growth.pdf
          12 kB
          Tim Armstrong

          Issue Links

            Activity

              People

              • Assignee:
                sailesh Sailesh Mukil
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: