Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
ghx-label-3
Description
Currently admission controls don't have a globally consistent view of how many queries have admitted so different coordinators can over-admit if queries arrive at multiple coordinators in-between topic updates. Topic updates happen every 2s by default. Worse, admission control topic updates can get stuck behind large catalog updates.
IMPALA-3649 proposes having a globally consistent view of admission control, which will require architectural changes.
In the meantime we could reduce the problem by increasing the frequency of topic updates for admission control and separating them from the larger catalog updates.
Attachments
Issue Links
- is related to
-
IMPALA-6583 Various tests fail with missing database or table from catalog
- Resolved
-
IMPALA-6785 Starting an Impalad on an already running cluster may result in inconsistent cluster subscription
- Resolved
-
IMPALA-4140 Increase default threadpool size for statestore heartbeat and topic update
- Resolved
- relates to
-
IMPALA-3649 Globally consistent admission control
- Open
Commit b0d3433e36d7942b3e10bddc310287266240810b in impala's branch refs/heads/master from tarmstrong@cloudera.com
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=b0d3433 ]
IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updatesThis adds a set of "prioritized" statestore topics that are small but
are important to deliver in a timely manner. These are delivered more
frequently by a separate thread pool to reduce the window for stale
admission control and scheduling information.
The contract between statestore and subscriber is changed so that the
statestore can send concurrent Update() RPCs for disjoint sets of
topics. This required changes to the subscriber implementation, which
assumed that only one Update RPC would arrive at a time.
It also changes the locking in the statestore so that the prioritized
update threads don't get stuck behind the catalog threads holding
'topic_lock_'. Specifically, it uses a reader-writer lock to protect
modification of the set of topics and a reader-writer lock per topic to
allow the topic data to be read by multiple threads concurrently.
Added metrics to monitor the per-topic update interval.
Testing:
Ran core tests.
Inspected metrics on Impala daemons, saw that membership and request
queue processing times had more samples recorded than the catalog
topic, reflecting the increased frequency.
Ran under thread sanitizer, made sure no data races were reported in
Statestore or StatestoreSubscriber.
Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Reviewed-on: http://gerrit.cloudera.org:8080/9123
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins