Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5746

Remote fragments continue to hold onto memory after stopping the coordinator daemon

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: None
    • Component/s: Distributed Exec
    • Labels:
      None

      Description

      Repro

      1. Start running queries
      2. Kill the coordinator node
      3. On the running Impalad check the memz tab, remote fragments continue to run and hold on to resources

      Remote fragments held on to memory +30 minutes after stopping the coordinator service.

      Attached thread dump from an Impalad running remote fragments .

      Snapshot of memz tab 30 minutes after killing the coordinator

      Process: Limit=201.73 GB Total=5.32 GB Peak=179.36 GB
        Free Disk IO Buffers: Total=1.87 GB Peak=1.87 GB
        RequestPool=root.default: Total=1.35 GB Peak=178.51 GB
          Query(f64169d4bb3c901c:3a21d8ae00000000): Total=2.64 MB Peak=104.73 MB
            Fragment f64169d4bb3c901c:3a21d8ae00000051: Total=2.64 MB Peak=2.67 MB
              AGGREGATION_NODE (id=15): Total=2.54 MB Peak=2.57 MB
                Exprs: Total=30.12 KB Peak=30.12 KB
              EXCHANGE_NODE (id=14): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=12.29 KB
              DataStreamSender (dst_id=17): Total=85.31 KB Peak=85.31 KB
              CodeGen: Total=1.53 KB Peak=374.50 KB
            Block Manager: Limit=161.39 GB Total=512.00 KB Peak=1.54 MB
          Query(2a4f12b3b4b1dc8c:db7e8cf200000000): Total=258.29 MB Peak=412.98 MB
            Fragment 2a4f12b3b4b1dc8c:db7e8cf20000008c: Total=2.29 MB Peak=2.29 MB
              SORT_NODE (id=11): Total=4.00 KB Peak=4.00 KB
              AGGREGATION_NODE (id=20): Total=2.27 MB Peak=2.27 MB
                Exprs: Total=25.12 KB Peak=25.12 KB
              EXCHANGE_NODE (id=19): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=0
              DataStreamSender (dst_id=21): Total=3.88 KB Peak=3.88 KB
              CodeGen: Total=4.17 KB Peak=1.05 MB
            Block Manager: Limit=161.39 GB Total=256.25 MB Peak=321.66 MB
          Query(68421d2a5dea0775:83f5d97200000000): Total=282.77 MB Peak=443.53 MB
            Fragment 68421d2a5dea0775:83f5d9720000004a: Total=26.77 MB Peak=26.92 MB
              SORT_NODE (id=8): Total=8.00 KB Peak=8.00 KB
                Exprs: Total=4.00 KB Peak=4.00 KB
              ANALYTIC_EVAL_NODE (id=7): Total=4.00 KB Peak=4.00 KB
                Exprs: Total=4.00 KB Peak=4.00 KB
              SORT_NODE (id=6): Total=24.00 MB Peak=24.00 MB
              AGGREGATION_NODE (id=12): Total=2.72 MB Peak=2.83 MB
                Exprs: Total=85.12 KB Peak=85.12 KB
              EXCHANGE_NODE (id=11): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=84.80 KB
              DataStreamSender (dst_id=13): Total=1.27 KB Peak=1.27 KB
              CodeGen: Total=24.80 KB Peak=4.13 MB
            Block Manager: Limit=161.39 GB Total=280.50 MB Peak=286.52 MB
          Query(e94c89fa89a74d27:82812bf900000000): Total=258.29 MB Peak=436.85 MB
            Fragment e94c89fa89a74d27:82812bf90000008e: Total=2.29 MB Peak=2.29 MB
              SORT_NODE (id=11): Total=4.00 KB Peak=4.00 KB
              AGGREGATION_NODE (id=20): Total=2.27 MB Peak=2.27 MB
                Exprs: Total=25.12 KB Peak=25.12 KB
              EXCHANGE_NODE (id=19): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=0
              DataStreamSender (dst_id=21): Total=3.88 KB Peak=3.88 KB
              CodeGen: Total=4.17 KB Peak=1.05 MB
            Block Manager: Limit=161.39 GB Total=256.25 MB Peak=321.62 MB
          Query(4e43dad3bdc935d8:938b8b7e00000000): Total=2.65 MB Peak=105.60 MB
            Fragment 4e43dad3bdc935d8:938b8b7e00000052: Total=2.65 MB Peak=2.68 MB
              AGGREGATION_NODE (id=15): Total=2.55 MB Peak=2.57 MB
                Exprs: Total=30.12 KB Peak=30.12 KB
              EXCHANGE_NODE (id=14): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=13.68 KB
              DataStreamSender (dst_id=17): Total=91.41 KB Peak=91.41 KB
              CodeGen: Total=1.53 KB Peak=374.50 KB
            Block Manager: Limit=161.39 GB Total=512.00 KB Peak=1.30 MB
          Query(b34bdd65f1ed017e:5a0291bd00000000): Total=2.37 MB Peak=106.56 MB
            Fragment b34bdd65f1ed017e:5a0291bd0000004b: Total=2.37 MB Peak=2.37 MB
              SORT_NODE (id=6): Total=4.00 KB Peak=4.00 KB
              AGGREGATION_NODE (id=10): Total=2.35 MB Peak=2.35 MB
                Exprs: Total=34.12 KB Peak=34.12 KB
              EXCHANGE_NODE (id=9): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=4.23 KB
              DataStreamSender (dst_id=11): Total=3.45 KB Peak=3.45 KB
              CodeGen: Total=4.51 KB Peak=1.11 MB
            Block Manager: Limit=161.39 GB Total=256.00 KB Peak=912.81 KB
          Query(b74ba58d53b6c45f:3e8228600000000): Total=190.41 MB Peak=425.09 MB
            Fragment b74ba58d53b6c45f:3e822860000009f: Total=67.90 KB Peak=2.34 MB
              SORT_NODE (id=14): Total=4.00 KB Peak=4.00 KB
              HASH_JOIN_NODE (id=13): Total=42.25 KB Peak=42.25 KB
                Exprs: Total=9.12 KB Peak=9.12 KB
                Hash Join Builder (join_node_id=13): Total=9.12 KB Peak=9.12 KB
                  Hash Join Builder (join_node_id=13) Exprs: Total=9.12 KB Peak=9.12 KB
              HDFS_SCAN_NODE (id=11): Total=0 Peak=0
              EXCHANGE_NODE (id=24): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=0
              DataStreamSender (dst_id=25): Total=1.05 KB Peak=1.05 KB
              CodeGen: Total=12.59 KB Peak=2.29 MB
            Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
            Fragment b74ba58d53b6c45f:3e8228600000085: Total=2.32 MB Peak=2.32 MB
              AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
                Exprs: Total=44.12 KB Peak=44.12 KB
              EXCHANGE_NODE (id=20): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=0
              DataStreamSender (dst_id=23): Total=22.09 KB Peak=22.09 KB
              CodeGen: Total=2.37 KB Peak=546.00 KB
            Fragment b74ba58d53b6c45f:3e8228600000060: Total=188.02 MB Peak=188.34 MB
              Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
              AGGREGATION_NODE (id=9): Total=1.67 MB Peak=1.67 MB
                Exprs: Total=44.12 KB Peak=44.12 KB
              HASH_JOIN_NODE (id=8): Total=1.13 MB Peak=1.15 MB
                Exprs: Total=9.12 KB Peak=9.12 KB
                Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
                  Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12 KB
              HASH_JOIN_NODE (id=7): Total=169.14 MB Peak=169.14 MB
                Exprs: Total=9.12 KB Peak=9.12 KB
                Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
                  Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12 KB
              EXCHANGE_NODE (id=17): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=587.50 KB
              EXCHANGE_NODE (id=18): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=316.11 KB
              EXCHANGE_NODE (id=19): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=4.70 KB
              DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
              CodeGen: Total=16.80 KB Peak=2.83 MB
          Query(cb4c14997ad6add2:c8f120100000000): Total=190.36 MB Peak=443.00 MB
            Fragment cb4c14997ad6add2:c8f1201000000a4: Total=67.90 KB Peak=2.34 MB
              SORT_NODE (id=14): Total=4.00 KB Peak=4.00 KB
              HASH_JOIN_NODE (id=13): Total=42.25 KB Peak=42.25 KB
                Exprs: Total=9.12 KB Peak=9.12 KB
                Hash Join Builder (join_node_id=13): Total=9.12 KB Peak=9.12 KB
                  Hash Join Builder (join_node_id=13) Exprs: Total=9.12 KB Peak=9.12 KB
              HDFS_SCAN_NODE (id=11): Total=0 Peak=0
              EXCHANGE_NODE (id=24): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=0
              DataStreamSender (dst_id=25): Total=1.05 KB Peak=1.05 KB
              CodeGen: Total=12.59 KB Peak=2.29 MB
            Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
            Fragment cb4c14997ad6add2:c8f120100000088: Total=2.33 MB Peak=2.33 MB
              AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
                Exprs: Total=44.12 KB Peak=44.12 KB
              EXCHANGE_NODE (id=20): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=0
              DataStreamSender (dst_id=23): Total=26.83 KB Peak=26.83 KB
              CodeGen: Total=2.37 KB Peak=546.00 KB
            Fragment cb4c14997ad6add2:c8f120100000063: Total=187.97 MB Peak=188.08 MB
              Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
              AGGREGATION_NODE (id=9): Total=1.67 MB Peak=1.67 MB
                Exprs: Total=44.12 KB Peak=44.12 KB
              HASH_JOIN_NODE (id=8): Total=1.14 MB Peak=1.15 MB
                Exprs: Total=9.12 KB Peak=9.12 KB
                Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
                  Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12 KB
              HASH_JOIN_NODE (id=7): Total=169.07 MB Peak=169.14 MB
                Exprs: Total=9.12 KB Peak=9.12 KB
                Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
                  Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12 KB
              EXCHANGE_NODE (id=17): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=314.15 KB
              EXCHANGE_NODE (id=18): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=861.18 KB
              EXCHANGE_NODE (id=19): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=4.70 KB
              DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
              CodeGen: Total=16.80 KB Peak=2.83 MB
          Query(f04a57ce97102dd7:c2a1081700000000): Total=190.31 MB Peak=419.11 MB
            Fragment f04a57ce97102dd7:c2a1081700000085: Total=2.33 MB Peak=2.33 MB
              AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
                Exprs: Total=44.12 KB Peak=44.12 KB
              EXCHANGE_NODE (id=20): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=0
              DataStreamSender (dst_id=23): Total=23.67 KB Peak=23.67 KB
              CodeGen: Total=2.37 KB Peak=546.00 KB
            Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
            Fragment f04a57ce97102dd7:c2a1081700000060: Total=187.99 MB Peak=188.07 MB
              Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
              AGGREGATION_NODE (id=9): Total=1.68 MB Peak=1.68 MB
                Exprs: Total=44.12 KB Peak=44.12 KB
              HASH_JOIN_NODE (id=8): Total=1.14 MB Peak=1.15 MB
                Exprs: Total=9.12 KB Peak=9.12 KB
                Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
                  Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12 KB
              HASH_JOIN_NODE (id=7): Total=169.09 MB Peak=169.14 MB
                Exprs: Total=9.12 KB Peak=9.12 KB
                Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
                  Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12 KB
              EXCHANGE_NODE (id=17): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=156.71 KB
              EXCHANGE_NODE (id=18): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=1.32 MB
              EXCHANGE_NODE (id=19): Total=0 Peak=0
              DataStreamRecvr: Total=0 Peak=4.70 KB
              DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
              CodeGen: Total=16.80 KB Peak=2.83 MB
        Untracked Memory: Total=2.10 GB
      

        Attachments

        1. remote_fragments_holding_memory.txt
          4.16 MB
          Mostafa Mokhtar

          Issue Links

            Activity

              People

              • Assignee:
                joemcdonnell Joe McDonnell
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: