Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6437

increase frequency of admission control topic updates

Details

    Description

      Currently admission controls don't have a globally consistent view of how many queries have admitted so different coordinators can over-admit if queries arrive at multiple coordinators in-between topic updates. Topic updates happen every 2s by default. Worse, admission control topic updates can get stuck behind large catalog updates.

      IMPALA-3649 proposes having a globally consistent view of admission control, which will require architectural changes.

      In the meantime we could reduce the problem by increasing the frequency of topic updates for admission control and separating them from the larger catalog updates.

      Attachments

        Issue Links

          Activity

            Commit b0d3433e36d7942b3e10bddc310287266240810b in impala's branch refs/heads/master from tarmstrong@cloudera.com
            [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=b0d3433 ]

            IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

            This adds a set of "prioritized" statestore topics that are small but
            are important to deliver in a timely manner. These are delivered more
            frequently by a separate thread pool to reduce the window for stale
            admission control and scheduling information.

            The contract between statestore and subscriber is changed so that the
            statestore can send concurrent Update() RPCs for disjoint sets of
            topics. This required changes to the subscriber implementation, which
            assumed that only one Update RPC would arrive at a time.

            It also changes the locking in the statestore so that the prioritized
            update threads don't get stuck behind the catalog threads holding
            'topic_lock_'. Specifically, it uses a reader-writer lock to protect
            modification of the set of topics and a reader-writer lock per topic to
            allow the topic data to be read by multiple threads concurrently.

            Added metrics to monitor the per-topic update interval.

            Testing:
            Ran core tests.

            Inspected metrics on Impala daemons, saw that membership and request
            queue processing times had more samples recorded than the catalog
            topic, reflecting the increased frequency.

            Ran under thread sanitizer, made sure no data races were reported in
            Statestore or StatestoreSubscriber.

            Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
            Reviewed-on: http://gerrit.cloudera.org:8080/9123
            Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
            Tested-by: Impala Public Jenkins

            jira-bot ASF subversion and git services added a comment - Commit b0d3433e36d7942b3e10bddc310287266240810b in impala's branch refs/heads/master from tarmstrong@cloudera.com [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=b0d3433 ] IMPALA-4953 , IMPALA-6437 : separate AC/scheduler from catalog topic updates This adds a set of "prioritized" statestore topics that are small but are important to deliver in a timely manner. These are delivered more frequently by a separate thread pool to reduce the window for stale admission control and scheduling information. The contract between statestore and subscriber is changed so that the statestore can send concurrent Update() RPCs for disjoint sets of topics. This required changes to the subscriber implementation, which assumed that only one Update RPC would arrive at a time. It also changes the locking in the statestore so that the prioritized update threads don't get stuck behind the catalog threads holding 'topic_lock_'. Specifically, it uses a reader-writer lock to protect modification of the set of topics and a reader-writer lock per topic to allow the topic data to be read by multiple threads concurrently. Added metrics to monitor the per-topic update interval. Testing: Ran core tests. Inspected metrics on Impala daemons, saw that membership and request queue processing times had more samples recorded than the catalog topic, reflecting the increased frequency. Ran under thread sanitizer, made sure no data races were reported in Statestore or StatestoreSubscriber. Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210 Reviewed-on: http://gerrit.cloudera.org:8080/9123 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins

            Commit 26309141d066706b07ecfb3572b5bc87eebfc736 in impala's branch refs/heads/2.x from tarmstrong@cloudera.com
            [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=2630914 ]

            IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

            This adds a set of "prioritized" statestore topics that are small but
            are important to deliver in a timely manner. These are delivered more
            frequently by a separate thread pool to reduce the window for stale
            admission control and scheduling information.

            The contract between statestore and subscriber is changed so that the
            statestore can send concurrent Update() RPCs for disjoint sets of
            topics. This required changes to the subscriber implementation, which
            assumed that only one Update RPC would arrive at a time.

            It also changes the locking in the statestore so that the prioritized
            update threads don't get stuck behind the catalog threads holding
            'topic_lock_'. Specifically, it uses a reader-writer lock to protect
            modification of the set of topics and a reader-writer lock per topic to
            allow the topic data to be read by multiple threads concurrently.

            Added metrics to monitor the per-topic update interval.

            Testing:
            Ran core tests.

            Inspected metrics on Impala daemons, saw that membership and request
            queue processing times had more samples recorded than the catalog
            topic, reflecting the increased frequency.

            Ran under thread sanitizer, made sure no data races were reported in
            Statestore or StatestoreSubscriber.

            Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
            Reviewed-on: http://gerrit.cloudera.org:8080/9123
            Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
            Tested-by: Impala Public Jenkins

            jira-bot ASF subversion and git services added a comment - Commit 26309141d066706b07ecfb3572b5bc87eebfc736 in impala's branch refs/heads/2.x from tarmstrong@cloudera.com [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=2630914 ] IMPALA-4953 , IMPALA-6437 : separate AC/scheduler from catalog topic updates This adds a set of "prioritized" statestore topics that are small but are important to deliver in a timely manner. These are delivered more frequently by a separate thread pool to reduce the window for stale admission control and scheduling information. The contract between statestore and subscriber is changed so that the statestore can send concurrent Update() RPCs for disjoint sets of topics. This required changes to the subscriber implementation, which assumed that only one Update RPC would arrive at a time. It also changes the locking in the statestore so that the prioritized update threads don't get stuck behind the catalog threads holding 'topic_lock_'. Specifically, it uses a reader-writer lock to protect modification of the set of topics and a reader-writer lock per topic to allow the topic data to be read by multiple threads concurrently. Added metrics to monitor the per-topic update interval. Testing: Ran core tests. Inspected metrics on Impala daemons, saw that membership and request queue processing times had more samples recorded than the catalog topic, reflecting the increased frequency. Ran under thread sanitizer, made sure no data races were reported in Statestore or StatestoreSubscriber. Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210 Reviewed-on: http://gerrit.cloudera.org:8080/9123 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins

            People

              tarmstrong Tim Armstrong
              tarmstrong Tim Armstrong
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: