Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9712

Hit OOM on TPC-H Q19

    XMLWordPrintableJSON

    Details

    • Epic Color:
      ghx-label-10

      Description

      We saw an out-of-memory on TPC-H 1TB with mt_dop=16.

          Query Status: Memory limit exceeded: ParquetColumnChunkReader::ReadDataPage() failed to allocate 65454 bytes for decompressed data.
      HDFS_SCAN_NODE (id=0) could not allocate 63.92 KB without exceeding limit.
      Error occurred on backend impala-executor-000-1:22000 by fragment 4b497d5e4237002a:94c50a5200000041
      Memory left in process limit: 71.74 GB
      Memory left in query limit: 52.96 KB
      Query(4b497d5e4237002a:94c50a5200000000): Limit=2.35 GB Reservation=1.56 GB ReservationLimit=1.88 GB OtherMemory=801.69 MB Total=2.35 GB Peak=2.35 GB
        Runtime Filter Bank: Reservation=1.00 MB ReservationLimit=1.00 MB OtherMemory=0 Total=1.00 MB Peak=1.00 MB
        Fragment 4b497d5e4237002a:94c50a52000000a7: Reservation=0 OtherMemory=0 Total=0 Peak=51.91 MB
          HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=48.78 MB
          KrpcDataStreamSender (dst_id=4): Total=0 Peak=31.17 KB
        CodeGen: Total=7.82 KB Peak=2.52 MB
        Fragment 4b497d5e4237002a:94c50a52000000a8: Reservation=0 OtherMemory=0 Total=0 Peak=40.15 MB
          HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=36.84 MB
          KrpcDataStreamSender (dst_id=4): Total=0 Peak=31.17 KB
        Fragment 4b497d5e4237002a:94c50a52000000a9: Reservation=0 OtherMemory=0 Total=0 Peak=31.79 MB
          HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=28.78 MB
          KrpcDataStreamSender (dst_id=4): Total=0 Peak=31.17 KB
        Fragment 4b497d5e4237002a:94c50a52000000c3: Reservation=544.00 MB OtherMemory=21.12 KB Total=544.02 MB Peak=544.36 MB
          EXCHANGE_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=352.00 KB
            KrpcDeferredRpcs: Total=0 Peak=0
          Hash Join Builder (join_node_id=2): Reservation=544.00 MB OtherMemory=13.12 KB Total=544.01 MB Peak=544.02 MB
            Hash Join Builder (join_node_id=2) Exprs: Total=13.12 KB Peak=13.12 KB
        Fragment 4b497d5e4237002a:94c50a5200000041: Reservation=66.00 MB OtherMemory=51.15 MB Total=117.15 MB Peak=117.29 MB
          AGGREGATION_NODE (id=3): Total=43.83 MB Peak=49.86 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=6.04 MB Total=6.04 MB Peak=8.29 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a5200000042: Reservation=66.00 MB OtherMemory=52.06 MB Total=118.06 MB Peak=118.06 MB
          AGGREGATION_NODE (id=3): Total=50.63 MB Peak=50.63 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=57.12 KB Total=57.12 KB Peak=7.79 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        CodeGen: Total=4.06 KB Peak=778.50 KB
        Fragment 4b497d5e4237002a:94c50a5200000043: Reservation=66.00 MB OtherMemory=47.81 MB Total=113.81 MB Peak=116.01 MB
          AGGREGATION_NODE (id=3): Total=45.01 MB Peak=48.67 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=1.48 MB Total=1.48 MB Peak=7.79 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a5200000044: Reservation=66.00 MB OtherMemory=56.25 MB Total=122.25 MB Peak=122.25 MB
          AGGREGATION_NODE (id=3): Total=54.81 MB Peak=54.81 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=57.12 KB Total=57.12 KB Peak=8.23 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a5200000045: Reservation=66.00 MB OtherMemory=49.68 MB Total=115.68 MB Peak=115.68 MB
          AGGREGATION_NODE (id=3): Total=43.95 MB Peak=44.61 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=4.36 MB Total=4.36 MB Peak=8.29 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a5200000046: Reservation=66.00 MB OtherMemory=48.62 MB Total=114.62 MB Peak=115.70 MB
          AGGREGATION_NODE (id=3): Total=42.33 MB Peak=48.36 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=4.92 MB Total=4.92 MB Peak=8.86 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a5200000047: Reservation=66.00 MB OtherMemory=49.53 MB Total=115.53 MB Peak=115.53 MB
          AGGREGATION_NODE (id=3): Total=44.45 MB Peak=47.74 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=3.79 MB Total=3.79 MB Peak=8.23 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a5200000048: Reservation=66.00 MB OtherMemory=54.38 MB Total=120.38 MB Peak=120.38 MB
          AGGREGATION_NODE (id=3): Total=53.13 MB Peak=53.13 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=57.12 KB Total=57.12 KB Peak=8.29 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.17 MB Total=67.17 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a5200000049: Reservation=66.00 MB OtherMemory=47.81 MB Total=113.81 MB Peak=115.08 MB
          AGGREGATION_NODE (id=3): Total=45.45 MB Peak=47.73 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=1005.84 KB Total=1005.84 KB Peak=8.23 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a520000004a: Reservation=66.00 MB OtherMemory=50.08 MB Total=116.08 MB Peak=117.15 MB
          AGGREGATION_NODE (id=3): Total=44.95 MB Peak=47.73 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=5.10 MB Total=5.10 MB Peak=7.79 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=4.00 KB Total=66.00 MB Peak=67.42 MB
            Exprs: Total=4.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        CodeGen: Total=53.44 KB Peak=13.23 MB
        Fragment 4b497d5e4237002a:94c50a520000004b: Reservation=66.00 MB OtherMemory=47.81 MB Total=113.81 MB Peak=115.08 MB
          AGGREGATION_NODE (id=3): Total=45.95 MB Peak=47.74 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=685.92 KB Total=685.92 KB Peak=8.29 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.17 MB Total=67.17 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a520000004c: Reservation=66.00 MB OtherMemory=48.34 MB Total=114.34 MB Peak=117.29 MB
          AGGREGATION_NODE (id=3): Total=43.83 MB Peak=49.86 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=3.23 MB Total=3.23 MB Peak=8.29 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a520000004d: Reservation=66.00 MB OtherMemory=45.07 MB Total=111.07 MB Peak=115.08 MB
          AGGREGATION_NODE (id=3): Total=37.84 MB Peak=47.73 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=6.04 MB Total=6.04 MB Peak=7.79 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.17 MB Total=67.17 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a520000004e: Reservation=66.00 MB OtherMemory=45.81 MB Total=111.81 MB Peak=113.58 MB
          AGGREGATION_NODE (id=3): Total=43.95 MB Peak=46.24 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=685.92 KB Total=685.92 KB Peak=7.73 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.17 MB Total=67.17 MB Peak=67.33 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a520000004f: Reservation=66.00 MB OtherMemory=54.59 MB Total=120.59 MB Peak=120.59 MB
          AGGREGATION_NODE (id=3): Total=53.25 MB Peak=53.25 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=57.12 KB Total=57.12 KB Peak=8.29 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
        Fragment 4b497d5e4237002a:94c50a5200000050: Reservation=66.00 MB OtherMemory=51.47 MB Total=117.47 MB Peak=117.47 MB
          AGGREGATION_NODE (id=3): Total=46.45 MB Peak=48.23 MB
            NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
              Exprs: Total=4.00 KB Peak=4.00 KB
          HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=3.73 MB Total=3.73 MB Peak=8.86 MB
            Exprs: Total=41.12 KB Peak=41.12 KB
          HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
            Exprs: Total=8.00 KB Peak=8.00 KB
          KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
      

      The aggregation node is accumulating a lot of memory and it's a non-grouping agg that shouldn't accumulate much mem:

          AGGREGATION_NODE (id=3): Total=43.83 MB Peak=49.86 MB
      

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: