Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7457

Allow StateStore subscribers to filter keys by a prefix

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.1.0
    • Backend, Catalog
    • None
    • ghx-label-3

    Description

      StateStore subscriptions are currently all-or-nothing. However, in some cases it might be useful to filter a subscription. Two use cases come to mind in the near term:

      1) currently the catalogd publishes objects into a topic. In order to publish, it has to subscribe to that topic. That means that the statestore then "reflects" all of the info right back to the catalogd causing extra network and memory pressure, even though the catalogd ignores the updates.

      2) in order to implement cache invalidation on "fetch-on-demand" impalads, we want to publish version numbers of catalog objects without sending the whole object. In order to keep SYNC_DDL working, it's much more convenient to publish the version numbers on the same topic as the full objects. However, the "fetch-on-demand" impalads shouldn't waste bandwidth from the statestore with all of the detailed object info.

      Attachments

        Activity

          People

            tlipcon Todd Lipcon
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: