[IMPALA-10611] test_wide_row fails with 'Failed to allocate row batch'

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Epic Color: ghx-label-14
• Assignee: Riza Suminto
• Reporter: Abhishek Rawat
• Created: 25/Mar/21 18:45
• Updated: 27/Apr/21 13:55
• Resolved: 26/Apr/21 17:32

Description

It appears that we may not be accounting for memory properly somewhere and as a result we end up with allocations much more than reservations resulting in OOM error. It's also possible we may not be freeing memory properly (could be a leak or memory accumulation due to smart pointers etc).

It is also possible that something (table data?) changed in the test case and as a result the memory limit is not sufficient.

Error:

Messagequery_test/test_scanners.py:276: in test_wide_row     self.run_test_case('QueryTest/wide-row', new_vector) common/impala_test_suite.py:665: in run_test_case     result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:603: in __exec_in_impala     result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:923: in __execute_query     return impalad_client.execute(query, user=user) common/impala_connection.py:205: in execute     return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute     handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:365: in __execute_query     self.wait_for_finished(handle) beeswax/impala_beeswax.py:386: in wait_for_finished     raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E   ImpalaBeeswaxException: ImpalaBeeswaxException: E    Query aborted:Memory limit exceeded: Failed to allocate row batch E   EXCHANGE_NODE (id=1) could not allocate 16.00 MB without exceeding limit. E   Error occurred on backend ip-172-31-14-211:27000 E   Memory left in process limit: 9.09 GB E   Memory left in query limit: 9.95 MB E   Query(7647dab034192a59:e4f2972900000000): Limit=100.00 MB Reservation=40.00 MB ReservationLimit=68.00 MB OtherMemory=50.05 MB Total=90.05 MB Peak=90.05 MB E     Unclaimed reservations: Reservation=32.00 MB OtherMemory=0 Total=32.00 MB Peak=40.00 MB E     Fragment 7647dab034192a59:e4f2972900000001: Reservation=8.00 MB OtherMemory=50.04 MB Total=58.04 MB Peak=58.05 MB E       HDFS_SCAN_NODE (id=0): Reservation=8.00 MB OtherMemory=30.01 MB Total=38.01 MB Peak=58.03 MB E       KrpcDataStreamSender (dst_id=1): Total=9.84 KB Peak=9.84 KB E     CodeGen: Total=1004.00 B Peak=118.00 KB E     Fragment 7647dab034192a59:e4f2972900000000: Reservation=0 OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB E       EXCHANGE_NODE (id=1): Reservation=8.00 KB OtherMemory=0 Total=8.00 KB Peak=8.00 KB E         KrpcDeferredRpcs: Total=0 Peak=0 E       PLAN_ROOT_SINK: Total=0 Peak=0 E     CodeGen: Total=90.00 B Peak=65.00 KB

StackTrace:

query_test/test_scanners.py:276: in test_wide_row
    self.run_test_case('QueryTest/wide-row', new_vector)
common/impala_test_suite.py:665: in run_test_case
    result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
common/impala_test_suite.py:603: in __exec_in_impala
    result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:923: in __execute_query
    return impalad_client.execute(query, user=user)
common/impala_connection.py:205: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:365: in __execute_query
    self.wait_for_finished(handle)
beeswax/impala_beeswax.py:386: in wait_for_finished
    raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    Query aborted:Memory limit exceeded: Failed to allocate row batch
E   EXCHANGE_NODE (id=1) could not allocate 16.00 MB without exceeding limit.
E   Error occurred on backend ip-172-31-14-211:27000
E   Memory left in process limit: 9.09 GB
E   Memory left in query limit: 9.95 MB
E   Query(7647dab034192a59:e4f2972900000000): Limit=100.00 MB Reservation=40.00 MB ReservationLimit=68.00 MB OtherMemory=50.05 MB Total=90.05 MB Peak=90.05 MB
E     Unclaimed reservations: Reservation=32.00 MB OtherMemory=0 Total=32.00 MB Peak=40.00 MB
E     Fragment 7647dab034192a59:e4f2972900000001: Reservation=8.00 MB OtherMemory=50.04 MB Total=58.04 MB Peak=58.05 MB
E       HDFS_SCAN_NODE (id=0): Reservation=8.00 MB OtherMemory=30.01 MB Total=38.01 MB Peak=58.03 MB
E       KrpcDataStreamSender (dst_id=1): Total=9.84 KB Peak=9.84 KB
E     CodeGen: Total=1004.00 B Peak=118.00 KB
E     Fragment 7647dab034192a59:e4f2972900000000: Reservation=0 OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB
E       EXCHANGE_NODE (id=1): Reservation=8.00 KB OtherMemory=0 Total=8.00 KB Peak=8.00 KB
E         KrpcDeferredRpcs: Total=0 Peak=0
E       PLAN_ROOT_SINK: Total=0 Peak=0
E     CodeGen: Total=90.00 B Peak=65.00 KB