[IMPALA-7064] Local memory admitted metric does not get updated immediately

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 3.0
• Component/s: Backend
• Labels: admission-control
• Epic Color: ghx-label-9
• Assignee: Bikramjeet Vig
• Reporter: Bikramjeet Vig
• Created: 23/May/18 18:14
• Updated: 19/Jun/18 12:12

Description

Due to a delay in updating (host_mem_reserved_[coord_id]) memory reserved locally on a host in impala, multiple queries submitted in succession after every previous succeeds can result in queries being queued momentarily due to not enough memory being available. This happens because unlike pool stats that immediately update local metrics, the host level memory reserved metric is only updated on the next statestore update. This delay can cause the admission controller to believe that memory reserved for a previous query has not been released.

Repro steps:

start an impala cluster using:
start-impala-cluster.py  -impalad_args="-default_pool_max_requests=1 --queue_wait_timeout_ms=1000000 --mem_limit=5G --default_pool_mem_limit=10G --queue_wait_timeout_ms=1"

run queries in succession like this:

./bin/impala-shell.sh -q "set mem_limit=5G;set num_nodes=1;select * from functional.alltypesagg, (select 1) B limit 1;select * from functional.alltypesagg, (select 1) B limit 1;"

Due to queue wait timeout set to 1ms, you will notice that the second query fails.