Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8178

Tests failing with “Could not allocate memory while trying to increase reservation” on EC filesystem

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 3.2.0
    • Fix Version/s: Impala 3.2.0
    • Component/s: Backend
    • Labels:
    • Epic Color:
      ghx-label-6

      Description

      In tests run against an Erasure Coding filesystem, multiple tests failed with memory allocation errors.

      In total 10 tests failed:

      • query_test.test_scanners.TestParquet.test_decimal_encodings
      • query_test.test_scanners.TestTpchScanRangeLengths.test_tpch_scan_ranges
      • query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 0]
      • query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 1]
      • query_test.test_hbase_queries.TestHBaseQueries.test_hbase_scan_node
      • query_test.test_scanners.TestParquet.test_def_levels
      • query_test.test_scanners.TestTextSplitDelimiters.test_text_split_across_buffers_delimiterquery_test.test_hbase_queries.TestHBaseQueries.test_hbase_filters
      • query_test.test_hbase_queries.TestHBaseQueries.test_hbase_inline_views
      • query_test.test_hbase_queries.TestHBaseQueries.test_hbase_top_n

      The first failure looked like this on the client side:

      F query_test/test_scanners.py::TestParquet:)::test_decimal_encodings[protocol: beeswax | exec_option:

      Unknown macro: {'batch_size'}

      | table_format: parquet/none]
      query_test/test_scanners.py:717: in test_decimal_encodings
      self.run_test_case('QueryTest/parquet-decimal-formats', vector, unique_database)
      common/impala_test_suite.py:472: in run_test_case
      result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:699: in __execute_query
      return impalad_client.execute(query, user=user)
      common/impala_connection.py:174: in execute
      return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:183: in execute
      handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:360: in __execute_query
      self.wait_for_finished(handle)
      beeswax/impala_beeswax.py:381: in wait_for_finished
      raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E ImpalaBeeswaxException: ImpalaBeeswaxException:
      E Query aborted:ExecQueryFInstances rpc query_id=6e44c3c949a31be2:f973c7ff00000000 failed: Failed to get minimum memory reservation of 8.00 KB on daemon xxx.com:22001 for query 6e44c3c949a31be2:f973c7ff00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
      E Query(6e44c3c949a31be2:f973c7ff00000000) could not allocate 8.00 KB without exceeding limit.
      E Error occurred on backend xxx.com:22001
      E Memory left in process limit: 1.19 GB
      E Query(6e44c3c949a31be2:f973c7ff00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
      E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.

      On the server side log:

      I0207 18:25:19.329311 5562 impala-server.cc:1063] 6e44c3c949a31be2:f973c7ff00000000] Registered query query_id=6e44c3c949a31be2:f973c7ff00000000 session_id=93497065f69e9d01:8a3bd06faff3da5
      I0207 18:25:19.329434 5562 Frontend.java:1242] 6e44c3c949a31be2:f973c7ff00000000] Analyzing query: select score from decimal_stored_as_int32
      I0207 18:25:19.329583 5562 FeSupport.java:285] 6e44c3c949a31be2:f973c7ff00000000] Requesting prioritized load of table(s): test_decimal_encodings_28d99c0e.decimal_stored_as_int32
      I0207 18:25:30.776041 5562 Frontend.java:1282] 6e44c3c949a31be2:f973c7ff00000000] Analysis finished.
      I0207 18:25:35.919486 10418 admission-controller.cc:608] 6e44c3c949a31be2:f973c7ff00000000] Schedule for id=6e44c3c949a31be2:f973c7ff00000000 in pool_name=default-pool per_host_mem_estimate=16.02 MB PoolConfig: max_requests=-1 max_queued=200 max_mem=-1.00 B
      I0207 18:25:35.919528 10418 admission-controller.cc:613] 6e44c3c949a31be2:f973c7ff00000000] Stats: agg_num_running=2, agg_num_queued=0, agg_mem_reserved=24.13 MB, local_host(local_mem_admitted=1.99 GB, num_admitted_running=2, num_queued=0, backend_mem_reserved=8.06 MB)
      I0207 18:25:35.919549 10418 admission-controller.cc:645] 6e44c3c949a31be2:f973c7ff00000000] Admitted query id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:35.920532 10418 coordinator.cc:93] 6e44c3c949a31be2:f973c7ff00000000] Exec() query_id=6e44c3c949a31be2:f973c7ff00000000 stmt=select score from decimal_stored_as_int32
      I0207 18:25:35.930855 10418 coordinator.cc:359] 6e44c3c949a31be2:f973c7ff00000000] starting execution on 2 backends for query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:35.938108 21110 impala-internal-service.cc:50] 6e44c3c949a31be2:f973c7ff00000000] ExecQueryFInstances(): query_id=6e44c3c949a31be2:f973c7ff00000000 coord=xxx.com:22000 #instances=1
      I0207 18:25:36.037228 10571 query-state.cc:624] 6e44c3c949a31be2:f973c7ff00000000] Executing instance. instance_id=6e44c3c949a31be2:f973c7ff00000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=5
      I0207 18:25:48.149771 12581 coordinator-backend-state.cc:209] ExecQueryFInstances rpc query_id=6e44c3c949a31be2:f973c7ff00000000 failed: Failed to get minimum memory reservation of 8.00 KB on daemon xxx.com:22001 for query 6e44c3c949a31be2:f973c7ff00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
      Query(6e44c3c949a31be2:f973c7ff00000000) could not allocate 8.00 KB without exceeding limit.
      Query(6e44c3c949a31be2:f973c7ff00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
      I0207 18:25:48.149895 10418 coordinator.cc:373] 6e44c3c949a31be2:f973c7ff00000000] started execution on 2 backends for query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.152803 10418 coordinator.cc:527] 6e44c3c949a31be2:f973c7ff00000000] ExecState: query id=6e44c3c949a31be2:f973c7ff00000000 finstance=N/A on host=xxx.com (EXECUTING -> ERROR) status=ExecQueryFInstances rpc query_id=6e44c3c949a31be2:f973c7ff00000000 failed: Failed to get minimum memory reservation of 8.00 KB on daemon xxx.com:22001 for query 6e44c3c949a31be2:f973c7ff00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
      Query(6e44c3c949a31be2:f973c7ff00000000) could not allocate 8.00 KB without exceeding limit.
      Query(6e44c3c949a31be2:f973c7ff00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
      I0207 18:25:48.152827 10418 coordinator-backend-state.cc:453] 6e44c3c949a31be2:f973c7ff00000000] Sending CancelQueryFInstances rpc for query_id=6e44c3c949a31be2:f973c7ff00000000 backend=127.0.0.1:27000
      I0207 18:25:48.155086 12737 control-service.cc:168] CancelQueryFInstances(): query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.155109 12737 query-exec-mgr.cc:97] QueryState: query_id=6e44c3c949a31be2:f973c7ff00000000 refcnt=4
      I0207 18:25:48.155117 12737 query-state.cc:649] Cancel: query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.155129 12737 krpc-data-stream-mgr.cc:325] cancelling all streams for fragment_instance_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.155297 10418 coordinator.cc:687] 6e44c3c949a31be2:f973c7ff00000000] CancelBackends() query_id=6e44c3c949a31be2:f973c7ff00000000, tried to cancel 1 backends
      I0207 18:25:48.155306 10418 coordinator.cc:859] 6e44c3c949a31be2:f973c7ff00000000] Release admission control resources for query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.170018 10571 krpc-data-stream-mgr.cc:294] 6e44c3c949a31be2:f973c7ff00000000] DeregisterRecvr(): fragment_instance_id=6e44c3c949a31be2:f973c7ff00000000, node=1
      I0207 18:25:48.197767 5562 impala-beeswax-server.cc:239] close(): query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.197775 5562 impala-server.cc:1142] UnregisterQuery(): query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.197779 5562 impala-server.cc:1249] Cancel(): query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.225905 10529 query-state.cc:272] 6e44c3c949a31be2:f973c7ff00000000] UpdateBackendExecState(): last report for 6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.225889 10571 query-state.cc:632] 6e44c3c949a31be2:f973c7ff00000000] Instance completed. instance_id=6e44c3c949a31be2:f973c7ff00000000 #in-flight=4 status=CANCELLED: Cancelled
      I0207 18:25:48.372977 12737 control-service.cc:125] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 6e44c3c949a31be2:f973c7ff00000000 remote host=127.0.0.1:50422
      I0207 18:25:48.373118 10529 query-state.cc:431] 6e44c3c949a31be2:f973c7ff00000000] Cancelling fragment instances as directed by the coordinator. Returned status: ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 6e44c3c949a31be2:f973c7ff00000000 remote host=127.0.0.1:50422
      I0207 18:25:48.373138 10529 query-state.cc:649] 6e44c3c949a31be2:f973c7ff00000000] Cancel: query_id=6e44c3c949a31be2:f973c7ff00000000
      I0207 18:25:48.429422 5562 query-exec-mgr.cc:184] ReleaseQueryState(): deleted query_id=6e44c3c949a31be2:f973c7ff00000000

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joemcdonnell Joe McDonnell
                Reporter:
                asherman Andrew Sherman
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: