Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6975

TestRuntimeRowFilters.test_row_filters failing with Memory limit exceeded

    XMLWordPrintableJSON

Details

    • ghx-label-3

    Description

      Error Message
query_test/test_runtime_filters.py:171: in test_row_filters     test_file_vars={'$RUNTIME_FILTER_WAIT_TIME_MS' : str(WAIT_TIME_MS)}) common/impala_test_suite.py:405: in run_test_case     result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:620: in __execute_query     return impalad_client.execute(query, user=user) common/impala_connection.py:160: in execute     return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:173: in execute     handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:341: in __execute_query     self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in wait_for_completion     raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E   ImpalaBeeswaxException: ImpalaBeeswaxException: E    Query aborted:Memory limit exceeded: ParquetColumnReader::ReadDataPage() failed to allocate 65533 bytes for decompressed data. E   HDFS_SCAN_NODE (id=0) could not allocate 64.00 KB without exceeding limit. E   Error occurred on backend impala-boost-static-burst-slave-el7-1aa3.vpc.cloudera.com:22001 by fragment f9423526590ed30b:732f1db100000001 E   Memory left in process limit: 11.14 GB E   Memory left in query limit: 22.16 KB E   Query(f9423526590ed30b:732f1db100000000): Limit=200.00 MB Reservation=160.00 MB ReservationLimit=160.00 MB OtherMemory=39.98 MB Total=199.98 MB Peak=199.98 MB E     Fragment f9423526590ed30b:732f1db100000007: Reservation=134.00 MB OtherMemory=30.22 MB Total=164.22 MB Peak=164.22 MB E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB E       AGGREGATION_NODE (id=3): Total=4.00 KB Peak=4.00 KB E         Exprs: Total=4.00 KB Peak=4.00 KB E       HASH_JOIN_NODE (id=2): Reservation=132.00 MB OtherMemory=198.25 KB Total=132.19 MB Peak=132.19 MB E         Exprs: Total=25.12 KB Peak=25.12 KB E         Hash Join Builder (join_node_id=2): Total=157.12 KB Peak=157.12 KB E           Hash Join Builder (join_node_id=2) Exprs: Total=149.12 KB Peak=149.12 KB E       EXCHANGE_NODE (id=4): Reservation=14.91 MB OtherMemory=67.76 KB Total=14.97 MB Peak=14.97 MB E         KrpcDeferredRpcs: Total=67.76 KB Peak=67.76 KB E       EXCHANGE_NODE (id=5): Reservation=15.03 MB OtherMemory=0 Total=15.03 MB Peak=15.03 MB E         KrpcDeferredRpcs: Total=0 Peak=45.12 KB E       KrpcDataStreamSender (dst_id=6): Total=16.00 KB Peak=16.00 KB E     Fragment f9423526590ed30b:732f1db100000001: Reservation=26.00 MB OtherMemory=9.75 MB Total=35.75 MB Peak=35.75 MB E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB E       HDFS_SCAN_NODE (id=0): Reservation=24.00 MB OtherMemory=9.30 MB Total=33.30 MB Peak=33.30 MB E         Exprs: Total=260.00 KB Peak=260.00 KB E       KrpcDataStreamSender (dst_id=4): Total=426.57 KB Peak=458.57 KB E         KrpcDataStreamSender (dst_id=4) Exprs: Total=256.00 KB Peak=256.00 KB E     Fragment f9423526590ed30b:732f1db100000004: Reservation=0 OtherMemory=0 Total=0 Peak=29.59 MB E       HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=29.32 MB E       KrpcDataStreamSender (dst_id=5): Total=0 Peak=266.57 KB E    E   Memory limit exceeded: ParquetColumnReader::ReadDataPage() failed to allocate 65533 bytes for decompressed data. E   HDFS_SCAN_NODE (id=0) could not allocate 64.00 KB without exceeding limit. E   Error occurred on backend impala-boost-static-burst-slave-el7-1aa3.vpc.cloudera.com:22001 by fragment f9423526590ed30b:732f1db100000001 E   Memory left in process limit: 11.14 GB E   Memory left in query limit: 22.16 KB E   Query(f9423526590ed30b:732f1db100000000): Limit=200.00 MB Reservation=160.00 MB ReservationLimit=160.00 MB OtherMemory=39.98 MB Total=199.98 MB Peak=199.98 MB E     Fragment f9423526590ed30b:732f1db100000007: Reservation=134.00 MB OtherMemory=30.22 MB Total=164.22 MB Peak=164.22 MB E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB E       AGGREGATION_NODE (id=3): Total=4.00 KB Peak=4.00 KB E         Exprs: Total=4.00 KB Peak=4.00 KB E       HASH_JOIN_NODE (id=2): Reservation=132.00 MB OtherMemory=198.25 KB Total=132.19 MB Peak=132.19 MB E         Exprs: Total=25.12 KB Peak=25.12 KB E         Hash Join Builder (join_node_id=2): Total=157.12 KB Peak=157.12 KB E           Hash Join Builder (join_node_id=2) Exprs: Total=149.12 KB Peak=149.12 KB E       EXCHANGE_NODE (id=4): Reservation=14.91 MB OtherMemory=67.76 KB Total=14.97 MB Peak=14.97 MB E         KrpcDeferredRpcs: Total=67.76 KB Peak=67.76 KB E       EXCHANGE_NODE (id=5): Reservation=15.03 MB OtherMemory=0 Total=15.03 MB Peak=15.03 MB E         KrpcDeferredRpcs: Total=0 Peak=45.12 KB E       KrpcDataStreamSender (dst_id=6): Total=16.00 KB Peak=16.00 KB E     Fragment f9423526590ed30b:732f1db100000001: Reservation=26.00 MB OtherMemory=9.75 MB Total=35.75 MB Peak=35.75 MB E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB E       HDFS_SCAN_NODE (id=0): Reservation=24.00 MB OtherMemory=9.30 MB Total=33.30 MB Peak=33.30 MB E         Exprs: Total=260.00 KB Peak=260.00 KB E       KrpcDataStreamSender (dst_id=4): Total=426.57 KB Peak=458.57 KB E         KrpcDataStreamSender (dst_id=4) Exprs: Total=256.00 KB Peak=256.00 KB E     Fragment f9423526590ed30b:732f1db100000004: Reservation=0 OtherMemory=0 Total=0 Peak=29.59 MB E       HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=29.32 MB E       KrpcDataStreamSender (dst_id=5): Total=0 Peak=266.57 KB
Stacktrace
query_test/test_runtime_filters.py:171: in test_row_filters
    test_file_vars={'$RUNTIME_FILTER_WAIT_TIME_MS' : str(WAIT_TIME_MS)})
common/impala_test_suite.py:405: in run_test_case
    result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:620: in __execute_query
    return impalad_client.execute(query, user=user)
common/impala_connection.py:160: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:173: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:341: in __execute_query
    self.wait_for_completion(handle)
beeswax/impala_beeswax.py:361: in wait_for_completion
    raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    Query aborted:Memory limit exceeded: ParquetColumnReader::ReadDataPage() failed to allocate 65533 bytes for decompressed data.
E   HDFS_SCAN_NODE (id=0) could not allocate 64.00 KB without exceeding limit.
E   Error occurred on backend impala-boost-static-burst-slave-el7-1aa3.vpc.cloudera.com:22001 by fragment f9423526590ed30b:732f1db100000001
E   Memory left in process limit: 11.14 GB
E   Memory left in query limit: 22.16 KB
E   Query(f9423526590ed30b:732f1db100000000): Limit=200.00 MB Reservation=160.00 MB ReservationLimit=160.00 MB OtherMemory=39.98 MB Total=199.98 MB Peak=199.98 MB
E     Fragment f9423526590ed30b:732f1db100000007: Reservation=134.00 MB OtherMemory=30.22 MB Total=164.22 MB Peak=164.22 MB
E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
E       AGGREGATION_NODE (id=3): Total=4.00 KB Peak=4.00 KB
E         Exprs: Total=4.00 KB Peak=4.00 KB
E       HASH_JOIN_NODE (id=2): Reservation=132.00 MB OtherMemory=198.25 KB Total=132.19 MB Peak=132.19 MB
E         Exprs: Total=25.12 KB Peak=25.12 KB
E         Hash Join Builder (join_node_id=2): Total=157.12 KB Peak=157.12 KB
E           Hash Join Builder (join_node_id=2) Exprs: Total=149.12 KB Peak=149.12 KB
E       EXCHANGE_NODE (id=4): Reservation=14.91 MB OtherMemory=67.76 KB Total=14.97 MB Peak=14.97 MB
E         KrpcDeferredRpcs: Total=67.76 KB Peak=67.76 KB
E       EXCHANGE_NODE (id=5): Reservation=15.03 MB OtherMemory=0 Total=15.03 MB Peak=15.03 MB
E         KrpcDeferredRpcs: Total=0 Peak=45.12 KB
E       KrpcDataStreamSender (dst_id=6): Total=16.00 KB Peak=16.00 KB
E     Fragment f9423526590ed30b:732f1db100000001: Reservation=26.00 MB OtherMemory=9.75 MB Total=35.75 MB Peak=35.75 MB
E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
E       HDFS_SCAN_NODE (id=0): Reservation=24.00 MB OtherMemory=9.30 MB Total=33.30 MB Peak=33.30 MB
E         Exprs: Total=260.00 KB Peak=260.00 KB
E       KrpcDataStreamSender (dst_id=4): Total=426.57 KB Peak=458.57 KB
E         KrpcDataStreamSender (dst_id=4) Exprs: Total=256.00 KB Peak=256.00 KB
E     Fragment f9423526590ed30b:732f1db100000004: Reservation=0 OtherMemory=0 Total=0 Peak=29.59 MB
E       HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=29.32 MB
E       KrpcDataStreamSender (dst_id=5): Total=0 Peak=266.57 KB
E   
E   Memory limit exceeded: ParquetColumnReader::ReadDataPage() failed to allocate 65533 bytes for decompressed data.
E   HDFS_SCAN_NODE (id=0) could not allocate 64.00 KB without exceeding limit.
E   Error occurred on backend impala-boost-static-burst-slave-el7-1aa3.vpc.cloudera.com:22001 by fragment f9423526590ed30b:732f1db100000001
E   Memory left in process limit: 11.14 GB
E   Memory left in query limit: 22.16 KB
E   Query(f9423526590ed30b:732f1db100000000): Limit=200.00 MB Reservation=160.00 MB ReservationLimit=160.00 MB OtherMemory=39.98 MB Total=199.98 MB Peak=199.98 MB
E     Fragment f9423526590ed30b:732f1db100000007: Reservation=134.00 MB OtherMemory=30.22 MB Total=164.22 MB Peak=164.22 MB
E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
E       AGGREGATION_NODE (id=3): Total=4.00 KB Peak=4.00 KB
E         Exprs: Total=4.00 KB Peak=4.00 KB
E       HASH_JOIN_NODE (id=2): Reservation=132.00 MB OtherMemory=198.25 KB Total=132.19 MB Peak=132.19 MB
E         Exprs: Total=25.12 KB Peak=25.12 KB
E         Hash Join Builder (join_node_id=2): Total=157.12 KB Peak=157.12 KB
E           Hash Join Builder (join_node_id=2) Exprs: Total=149.12 KB Peak=149.12 KB
E       EXCHANGE_NODE (id=4): Reservation=14.91 MB OtherMemory=67.76 KB Total=14.97 MB Peak=14.97 MB
E         KrpcDeferredRpcs: Total=67.76 KB Peak=67.76 KB
E       EXCHANGE_NODE (id=5): Reservation=15.03 MB OtherMemory=0 Total=15.03 MB Peak=15.03 MB
E         KrpcDeferredRpcs: Total=0 Peak=45.12 KB
E       KrpcDataStreamSender (dst_id=6): Total=16.00 KB Peak=16.00 KB
E     Fragment f9423526590ed30b:732f1db100000001: Reservation=26.00 MB OtherMemory=9.75 MB Total=35.75 MB Peak=35.75 MB
E       Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
E       HDFS_SCAN_NODE (id=0): Reservation=24.00 MB OtherMemory=9.30 MB Total=33.30 MB Peak=33.30 MB
E         Exprs: Total=260.00 KB Peak=260.00 KB
E       KrpcDataStreamSender (dst_id=4): Total=426.57 KB Peak=458.57 KB
E         KrpcDataStreamSender (dst_id=4) Exprs: Total=256.00 KB Peak=256.00 KB
E     Fragment f9423526590ed30b:732f1db100000004: Reservation=0 OtherMemory=0 Total=0 Peak=29.59 MB
E       HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=29.32 MB
E       KrpcDataStreamSender (dst_id=5): Total=0 Peak=266.57 KB

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue IMPALA-6594 Various tests failing (some flaky) with memory limit exceeded

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              sailesh Sailesh Mukil
              sailesh Sailesh Mukil
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: