Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6792

Appears to be a memory leak in orphaned fragments

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
    • Impala 2.12.0
    • Backend
    • None

    Description

      On Impala daemons that have orphaned fragments it appears that memory continued to grow after query got cancelled, indicating of a memory leak.
      Memz page indicate that some fragments continue to run.

      Memory Usage
Memory consumption / limit: 101.48 GB / 64.00 GB
Breakdown
Process: memory limit exceeded. Limit=64.00 GB Total=101.51 GB Peak=101.51 GB
  Buffer Pool: Free Buffers: Total=0
  Buffer Pool: Clean Pages: Total=0
  Buffer Pool: Unused Reservation: Total=-71.09 MB
  Data Stream Service Queue: Limit=3.20 GB Total=0 Peak=875.26 KB
  Data Stream Manager Early RPCs: Total=0 Peak=23.03 MB
  TCMalloc Overhead: Total=760.53 MB
  Free Disk IO Buffers: Total=1.24 GB Peak=1.30 GB
  RequestPool=root.default: Total=909.59 MB Peak=11.91 GB
    Query(6c4aa5355486f3f6:5125570400000000): Limit=6.00 GB Reservation=69.50 MB ReservationLimit=4.80 GB OtherMemory=67.90 MB Total=137.40 MB Peak=400.25 MB
      Unclaimed reservations: Reservation=34.00 MB OtherMemory=0 Total=34.00 MB Peak=90.14 MB
      Fragment 6c4aa5355486f3f6:5125570400000845: Reservation=35.50 MB OtherMemory=31.87 MB Total=67.37 MB Peak=67.37 MB
        Runtime Filter Bank: Reservation=512.00 KB ReservationLimit=512.00 KB OtherMemory=0 Total=512.00 KB Peak=512.00 KB
        AGGREGATION_NODE (id=12): Total=4.00 KB Peak=4.00 KB
          Exprs: Total=4.00 KB Peak=4.00 KB
        AGGREGATION_NODE (id=11): Total=17.12 KB Peak=17.12 KB
          Exprs: Total=17.12 KB Peak=17.12 KB
        HASH_JOIN_NODE (id=10): Reservation=17.00 MB OtherMemory=66.25 KB Total=17.06 MB Peak=17.06 MB
          Exprs: Total=13.12 KB Peak=13.12 KB
          Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=13.12 KB
            Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB Peak=13.12 KB
        EXCHANGE_NODE (id=19): Reservation=10.59 MB OtherMemory=6.69 MB Total=17.29 MB Peak=17.29 MB
          KrpcDeferredRpcs: Total=6.69 MB Peak=6.69 MB
        HASH_JOIN_NODE (id=9): Reservation=18.00 MB OtherMemory=34.25 KB Total=18.03 MB Peak=18.10 MB
          Exprs: Total=13.12 KB Peak=13.12 KB
          Hash Join Builder (join_node_id=9): Total=13.12 KB Peak=45.12 KB
            Hash Join Builder (join_node_id=9) Exprs: Total=13.12 KB Peak=13.12 KB
        EXCHANGE_NODE (id=17): Reservation=14.44 MB OtherMemory=0 Total=14.44 MB Peak=14.45 MB
          KrpcDeferredRpcs: Total=0 Peak=2.42 MB
        EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=776.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        KrpcDataStreamSender (dst_id=20): Total=3.12 KB Peak=3.12 KB
        CodeGen: Total=18.24 KB Peak=2.89 MB
      Fragment 6c4aa5355486f3f6:5125570400000047: Reservation=0 OtherMemory=0 Total=0 Peak=41.25 MB
        HDFS_SCAN_NODE (id=4): Total=0 Peak=18.75 MB
        KrpcDataStreamSender (dst_id=17): Total=0 Peak=22.19 MB
        CodeGen: Total=0 Peak=222.50 KB
      Fragment 6c4aa5355486f3f6:512557040000033f: Reservation=0 OtherMemory=0 Total=0 Peak=241.80 MB
        HASH_JOIN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=2.00 MB
          Hash Join Builder (join_node_id=8): Total=0 Peak=21.12 KB
        HASH_JOIN_NODE (id=7): Reservation=0 OtherMemory=0 Total=0 Peak=1.99 MB
          Hash Join Builder (join_node_id=7): Total=0 Peak=21.12 KB
        HASH_JOIN_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=128.08 MB
          Hash Join Builder (join_node_id=6): Total=0 Peak=21.12 KB
        HDFS_SCAN_NODE (id=0): Total=0 Peak=106.87 MB
        EXCHANGE_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=2.46 MB
          KrpcDeferredRpcs: Total=0 Peak=388.27 KB
        EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        KrpcDataStreamSender (dst_id=18): Total=0 Peak=2.61 MB
        CodeGen: Total=0 Peak=3.61 MB
      Fragment 6c4aa5355486f3f6:51255704000002e8: Reservation=0 OtherMemory=0 Total=0 Peak=6.71 MB
        HDFS_SCAN_NODE (id=2): Total=0 Peak=3.44 MB
        KrpcDataStreamSender (dst_id=14): Total=0 Peak=3.23 MB
        CodeGen: Total=0 Peak=241.50 KB
      Fragment 6c4aa5355486f3f6:51255704000005ea: Reservation=0 OtherMemory=36.02 MB Total=36.02 MB Peak=46.83 MB
        HDFS_SCAN_NODE (id=5): Total=9.55 MB Peak=18.17 MB
        KrpcDataStreamSender (dst_id=19): Total=26.46 MB Peak=28.87 MB
        CodeGen: Total=234.00 B Peak=52.50 KB
    Query(b44c5962ad16140:4bd217e100000000): Limit=6.00 GB Reservation=660.28 MB ReservationLimit=4.80 GB OtherMemory=111.92 MB Total=772.20 MB Peak=778.99 MB
      Unclaimed reservations: Reservation=3.94 MB OtherMemory=0 Total=3.94 MB Peak=120.66 MB
      Fragment b44c5962ad16140:4bd217e100000bbb: Reservation=0 OtherMemory=49.74 KB Total=49.74 KB Peak=1.41 MB
        SORT_NODE (id=16): Total=8.00 KB Peak=8.00 KB
          Exprs: Total=4.00 KB Peak=4.00 KB
        AGGREGATION_NODE (id=27): Total=25.12 KB Peak=25.12 KB
          Exprs: Total=25.12 KB Peak=25.12 KB
        EXCHANGE_NODE (id=26): Reservation=0 OtherMemory=0 Total=0 Peak=0
          KrpcDeferredRpcs: Total=0 Peak=0
        KrpcDataStreamSender (dst_id=28): Total=3.88 KB Peak=3.88 KB
        CodeGen: Total=4.74 KB Peak=1.37 MB
      Fragment b44c5962ad16140:4bd217e100000934: Reservation=332.39 MB OtherMemory=15.81 MB Total=348.20 MB Peak=358.59 MB
        Runtime Filter Bank: Reservation=2.52 MB ReservationLimit=2.52 MB OtherMemory=0 Total=2.52 MB Peak=2.52 MB
        AGGREGATION_NODE (id=15): Total=25.12 KB Peak=25.12 KB
          Exprs: Total=25.12 KB Peak=25.12 KB
        HASH_JOIN_NODE (id=14): Reservation=1.94 MB OtherMemory=82.25 KB Total=2.02 MB Peak=2.03 MB
          Exprs: Total=13.12 KB Peak=13.12 KB
          Hash Join Builder (join_node_id=14): Total=13.12 KB Peak=21.12 KB
            Hash Join Builder (join_node_id=14) Exprs: Total=13.12 KB Peak=13.12 KB
        HASH_JOIN_NODE (id=13): Reservation=1.94 MB OtherMemory=74.25 KB Total=2.01 MB Peak=2.03 MB
          Exprs: Total=13.12 KB Peak=13.12 KB
          Hash Join Builder (join_node_id=13): Total=13.12 KB Peak=21.12 KB
            Hash Join Builder (join_node_id=13) Exprs: Total=13.12 KB Peak=13.12 KB
        HASH_JOIN_NODE (id=12): Reservation=160.00 MB OtherMemory=134.25 KB Total=160.13 MB Peak=160.15 MB
          Exprs: Total=47.12 KB Peak=47.12 KB
          Hash Join Builder (join_node_id=12): Total=47.12 KB Peak=55.12 KB
            Hash Join Builder (join_node_id=12) Exprs: Total=47.12 KB Peak=47.12 KB
        HASH_JOIN_NODE (id=11): Reservation=160.00 MB OtherMemory=62.25 KB Total=160.06 MB Peak=160.08 MB
          Exprs: Total=17.12 KB Peak=17.12 KB
          Hash Join Builder (join_node_id=11): Total=13.12 KB Peak=21.12 KB
            Hash Join Builder (join_node_id=11) Exprs: Total=13.12 KB Peak=13.12 KB
        HASH_JOIN_NODE (id=10): Reservation=6.00 MB OtherMemory=54.25 KB Total=6.05 MB Peak=6.07 MB
          Exprs: Total=17.12 KB Peak=17.12 KB
          Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=21.12 KB
            Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB Peak=13.12 KB
        EXCHANGE_NODE (id=20): Reservation=14.34 MB OtherMemory=0 Total=14.34 MB Peak=14.34 MB
          KrpcDeferredRpcs: Total=0 Peak=1.16 MB
        EXCHANGE_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=2.11 MB
          KrpcDeferredRpcs: Total=0 Peak=232.95 KB
        EXCHANGE_NODE (id=22): Reservation=0 OtherMemory=0 Total=0 Peak=10.97 MB
          KrpcDeferredRpcs: Total=0 Peak=805.37 KB
        EXCHANGE_NODE (id=23): Reservation=0 OtherMemory=0 Total=0 Peak=13.84 MB
          KrpcDeferredRpcs: Total=0 Peak=1.69 MB
        EXCHANGE_NODE (id=24): Reservation=0 OtherMemory=0 Total=0 Peak=40.00 KB
          KrpcDeferredRpcs: Total=0 Peak=8.10 KB
        EXCHANGE_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        KrpcDataStreamSender (dst_id=26): Total=1021.05 KB Peak=1021.05 KB
        CodeGen: Total=48.00 KB Peak=8.65 MB
      Fragment b44c5962ad16140:4bd217e10000054e: Reservation=321.00 MB OtherMemory=20.82 MB Total=341.82 MB Peak=352.83 MB
        Runtime Filter Bank: Reservation=1.00 MB ReservationLimit=1.00 MB OtherMemory=0 Total=1.00 MB Peak=1.00 MB
        HASH_JOIN_NODE (id=9): Reservation=320.00 MB OtherMemory=60.25 KB Total=320.06 MB Peak=320.07 MB
          Exprs: Total=22.12 KB Peak=22.12 KB
          Hash Join Builder (join_node_id=9): Total=22.12 KB Peak=30.12 KB
            Hash Join Builder (join_node_id=9) Exprs: Total=22.12 KB Peak=22.12 KB
        EXCHANGE_NODE (id=18): Reservation=12.48 MB OtherMemory=0 Total=12.48 MB Peak=16.40 MB
          KrpcDeferredRpcs: Total=0 Peak=4.41 MB
        EXCHANGE_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=4.35 MB
          KrpcDeferredRpcs: Total=0 Peak=1.69 MB
        KrpcDataStreamSender (dst_id=20): Total=8.26 MB Peak=19.26 MB
        CodeGen: Total=8.02 KB Peak=1.34 MB
      Fragment b44c5962ad16140:4bd217e100000048: Reservation=2.95 MB OtherMemory=75.24 MB Total=78.19 MB Peak=86.44 MB
        Runtime Filter Bank: Reservation=1.02 MB ReservationLimit=1.02 MB OtherMemory=0 Total=1.02 MB Peak=1.02 MB
        HASH_JOIN_NODE (id=8): Reservation=1.94 MB OtherMemory=79.25 KB Total=2.01 MB Peak=2.01 MB
          Exprs: Total=13.12 KB Peak=13.12 KB
          Hash Join Builder (join_node_id=8): Total=13.12 KB Peak=21.12 KB
            Hash Join Builder (join_node_id=8) Exprs: Total=13.12 KB Peak=13.12 KB
        HDFS_SCAN_NODE (id=0): Total=63.00 MB Peak=63.01 MB
        EXCHANGE_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
          KrpcDeferredRpcs: Total=0 Peak=0
        KrpcDataStreamSender (dst_id=18): Total=12.09 MB Peak=20.43 MB
        CodeGen: Total=8.66 KB Peak=1.50 MB
      Fragment b44c5962ad16140:4bd217e1000002f3: Reservation=0 OtherMemory=0 Total=0 Peak=81.41 MB
        HDFS_SCAN_NODE (id=1): Total=0 Peak=60.83 MB
        KrpcDataStreamSender (dst_id=19): Total=0 Peak=19.08 MB
        CodeGen: Total=0 Peak=335.00 KB
      Fragment b44c5962ad16140:4bd217e1000007ef: Reservation=0 OtherMemory=0 Total=0 Peak=26.00 MB
        HDFS_SCAN_NODE (id=5): Total=0 Peak=6.29 MB
        KrpcDataStreamSender (dst_id=21): Total=0 Peak=19.66 MB
        CodeGen: Total=0 Peak=241.50 KB
  Untracked Memory: Total=98.71 GB
tcmalloc
------------------------------------------------
MALLOC:   108196856848 (103184.6 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +    625752432 (  596.8 MiB) Bytes in central cache freelist
MALLOC: +     25029120 (   23.9 MiB) Bytes in transfer cache freelist
MALLOC: +    146667648 (  139.9 MiB) Bytes in thread cache freelists
MALLOC: +    449097920 (  428.3 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: = 109443403968 (104373.4 MiB) Actual memory used (physical + swap)
MALLOC: +       221184 (    0.2 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: = 109443625152 (104373.6 MiB) Virtual address space used
MALLOC:
MALLOC:        6575796              Spans in use
MALLOC:            380              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
System
Physical Memory: 125.52 GB
Transparent Huge Pages Config:
  enabled: always [madvise] never
  defrag: always [madvise] never
  khugepaged defrag: 1
Process and system memory metrics
Name	Value	Description
memory.anon-huge-page-bytes	640.00 MB	Total bytes of anonymous (a.k.a. transparent) huge pages used by this process.
memory.mapped-bytes	137.22 GB	Total bytes of memory mappings in this process (the virtual memory size).
memory.num-maps	4236	Total number of memory mappings in this process.
memory.rss	68.40 GB	Resident set size (RSS) of this process, including TCMalloc, buffer pool and Jvm.
memory.thp.defrag	always [madvise] never	The system-wide 'defrag' setting for Transparent Huge Pages.
memory.thp.enabled	always [madvise] never	The system-wide 'enabled' setting for Transparent Huge Pages.
memory.thp.khugepaged-defrag	1	The system-wide 'defrag' setting for khugepaged.
memory.total-used	101.51 GB	Total memory currently used by TCMalloc and buffer pool.
Buffer pool memory metrics
Name	Value	Description
buffer-pool.clean-page-bytes	0	Total bytes of clean page memory cached in the buffer pool.
buffer-pool.clean-pages	0	Total number of clean pages cached in the buffer pool.
buffer-pool.clean-pages-limit	5.44 GB	Limit on number of clean pages cached in the buffer pool.
buffer-pool.free-buffer-bytes	0	Total bytes of free buffer memory cached in the buffer pool.
buffer-pool.free-buffers	0	Total number of free buffers cached in the buffer pool.
buffer-pool.limit	54.40 GB	Maximum allowed bytes allocated by the buffer pool.
buffer-pool.reserved	781.62 MB	Total bytes of buffers reserved by Impala subsystems
buffer-pool.system-allocated	710.53 MB	Total buffer memory currently allocated by the buffer pool.
buffer-pool.unused-reservation-bytes	71.09 MB	Total bytes of buffer reservations by Impala subsystems that are currently unused
JVM aggregate memory metrics
Name	Value	Description
jvm.total.committed-usage-bytes	1.73 GB	Jvm total Committed Usage Bytes
jvm.total.current-usage-bytes	385.19 MB	Jvm total Current Usage Bytes
jvm.total.init-usage-bytes	1.88 GB	Jvm total Init Usage Bytes
jvm.total.max-usage-bytes	31.23 GB	Jvm total Max Usage Bytes
jvm.total.peak-committed-usage-bytes	2.39 GB	Jvm total Peak Committed Usage Bytes
jvm.total.peak-current-usage-bytes	1.08 GB	Jvm total Peak Current Usage Bytes
jvm.total.peak-init-usage-bytes	1.88 GB	Jvm total Peak Init Usage Bytes
jvm.total.peak-max-usage-bytes	31.32 GB	Jvm total Peak Max Usage Bytes
JVM heap memory metrics
Name	Value	Description
jvm.heap.committed-usage-bytes	1.67 GB	Jvm heap Committed Usage Bytes
jvm.heap.current-usage-bytes	322.88 MB	Jvm heap Current Usage Bytes
jvm.heap.init-usage-bytes	1.96 GB	Jvm heap Init Usage Bytes
jvm.heap.max-usage-bytes	26.67 GB	Jvm heap Max Usage Bytes
jvm.heap.peak-committed-usage-bytes	0	Jvm heap Peak Committed Usage Bytes
jvm.heap.peak-current-usage-bytes	0	Jvm heap Peak Current Usage Bytes
jvm.heap.peak-init-usage-bytes	0	Jvm heap Peak Init Usage Bytes
jvm.heap.peak-max-usage-bytes	0	Jvm heap Peak Max Usage Bytes
JVM non-heap memory metrics
Name	Value	Description
jvm.non-heap.committed-usage-bytes	63.40 MB	Jvm non-heap Committed Usage Bytes
jvm.non-heap.current-usage-bytes	62.31 MB	Jvm non-heap Current Usage Bytes
jvm.non-heap.init-usage-bytes	2.44 MB	Jvm non-heap Init Usage Bytes
jvm.non-heap.max-usage-bytes	-1.00 B	Jvm non-heap Max Usage Bytes
jvm.non-heap.peak-committed-usage-bytes	0	Jvm non-heap Peak Committed Usage Bytes
jvm.non-heap.peak-current-usage-bytes	0	Jvm non-heap Peak Current Usage Bytes
jvm.non-heap.peak-init-usage-bytes	0	Jvm non-heap Peak Init Usage Bytes
jvm.non-heap.peak-max-usage-bytes	0	Jvm non-heap Peak Max Usage Bytes

      Attachments

        1. memory_leak.txt
          443 kB
          Mostafa Mokhtar
        2. Screen Shot 2018-04-02 at 12.08.54 PM.png
          141 kB
          Mostafa Mokhtar
        3. growth.pdf
          12 kB
          Tim Armstrong

        Issue Links

          is broken by

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5558 Query hang after coordinator crash because DoRpc(ReportExecStatus) fails and is not retried

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5576 Wrong Cancel() in QueryState::ReportExecStatusAux() can lead to coordinator hang

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-9919 Bad Impala Performance after a period of time

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. THRIFT-2225 SSLContext destroy before cleanupOpenSSL

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. THRIFT-2936 Minor memory leak in SSL

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              sailesh Sailesh Mukil
              mmokhtar Mostafa Mokhtar
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: