Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10611

test_wide_row fails with 'Failed to allocate row batch'

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Impala 4.0.0
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-14

      Description

      It appears that we may not be accounting for memory properly somewhere and as a result we end up with allocations much more than reservations resulting in OOM error. It's also possible we may not be freeing memory properly (could be a leak or memory accumulation due to smart pointers etc).

      It is also possible that something (table data?) changed in the test case and as a result the memory limit is not sufficient.

      Error:

      Messagequery_test/test_scanners.py:276: in test_wide_row     self.run_test_case('QueryTest/wide-row', new_vector) common/impala_test_suite.py:665: in run_test_case     result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:603: in __exec_in_impala     result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:923: in __execute_query     return impalad_client.execute(query, user=user) common/impala_connection.py:205: in execute     return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute     handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:365: in __execute_query     self.wait_for_finished(handle) beeswax/impala_beeswax.py:386: in wait_for_finished     raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E   ImpalaBeeswaxException: ImpalaBeeswaxException: E    Query aborted:Memory limit exceeded: Failed to allocate row batch E   EXCHANGE_NODE (id=1) could not allocate 16.00 MB without exceeding limit. E   Error occurred on backend ip-172-31-14-211:27000 E   Memory left in process limit: 9.09 GB E   Memory left in query limit: 9.95 MB E   Query(7647dab034192a59:e4f2972900000000): Limit=100.00 MB Reservation=40.00 MB ReservationLimit=68.00 MB OtherMemory=50.05 MB Total=90.05 MB Peak=90.05 MB E     Unclaimed reservations: Reservation=32.00 MB OtherMemory=0 Total=32.00 MB Peak=40.00 MB E     Fragment 7647dab034192a59:e4f2972900000001: Reservation=8.00 MB OtherMemory=50.04 MB Total=58.04 MB Peak=58.05 MB E       HDFS_SCAN_NODE (id=0): Reservation=8.00 MB OtherMemory=30.01 MB Total=38.01 MB Peak=58.03 MB E       KrpcDataStreamSender (dst_id=1): Total=9.84 KB Peak=9.84 KB E     CodeGen: Total=1004.00 B Peak=118.00 KB E     Fragment 7647dab034192a59:e4f2972900000000: Reservation=0 OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB E       EXCHANGE_NODE (id=1): Reservation=8.00 KB OtherMemory=0 Total=8.00 KB Peak=8.00 KB E         KrpcDeferredRpcs: Total=0 Peak=0 E       PLAN_ROOT_SINK: Total=0 Peak=0 E     CodeGen: Total=90.00 B Peak=65.00 KB

      StackTrace:

      query_test/test_scanners.py:276: in test_wide_row
          self.run_test_case('QueryTest/wide-row', new_vector)
      common/impala_test_suite.py:665: in run_test_case
          result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
      common/impala_test_suite.py:603: in __exec_in_impala
          result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:923: in __execute_query
          return impalad_client.execute(query, user=user)
      common/impala_connection.py:205: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:187: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:365: in __execute_query
          self.wait_for_finished(handle)
      beeswax/impala_beeswax.py:386: in wait_for_finished
          raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    Query aborted:Memory limit exceeded: Failed to allocate row batch
      E   EXCHANGE_NODE (id=1) could not allocate 16.00 MB without exceeding limit.
      E   Error occurred on backend ip-172-31-14-211:27000
      E   Memory left in process limit: 9.09 GB
      E   Memory left in query limit: 9.95 MB
      E   Query(7647dab034192a59:e4f2972900000000): Limit=100.00 MB Reservation=40.00 MB ReservationLimit=68.00 MB OtherMemory=50.05 MB Total=90.05 MB Peak=90.05 MB
      E     Unclaimed reservations: Reservation=32.00 MB OtherMemory=0 Total=32.00 MB Peak=40.00 MB
      E     Fragment 7647dab034192a59:e4f2972900000001: Reservation=8.00 MB OtherMemory=50.04 MB Total=58.04 MB Peak=58.05 MB
      E       HDFS_SCAN_NODE (id=0): Reservation=8.00 MB OtherMemory=30.01 MB Total=38.01 MB Peak=58.03 MB
      E       KrpcDataStreamSender (dst_id=1): Total=9.84 KB Peak=9.84 KB
      E     CodeGen: Total=1004.00 B Peak=118.00 KB
      E     Fragment 7647dab034192a59:e4f2972900000000: Reservation=0 OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB
      E       EXCHANGE_NODE (id=1): Reservation=8.00 KB OtherMemory=0 Total=8.00 KB Peak=8.00 KB
      E         KrpcDeferredRpcs: Total=0 Peak=0
      E       PLAN_ROOT_SINK: Total=0 Peak=0
      E     CodeGen: Total=90.00 B Peak=65.00 KB
      

        Attachments

          Activity

            People

            • Assignee:
              rizaon Riza Suminto
              Reporter:
              arawat Abhishek Rawat
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: