Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10265

Doc about enable_incremental_metadata_updates flag

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Documentation
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Docs
    • Labels:
      None
    • Epic Color:
      ghx-label-8

      Description

      IMPALA-10113 adds a feature flag to turn off the incremental metadata update feature which defaults to be turned on. This flag decides how catalogd propagates metadata updates to the catalog topic.

      If enable_incremental_metadata_updates is true, catalogd will send metadata updates in partition granularity. So a table that just has one partition changed will only have an update on that partition. This reduces the size of the metadata that needs to be sent from the catalogd.

      If enable_incremental_metadata_updates is false, that's the legacy behavior. Catalogd will send metadata updates in table granularity. So a table that just has one partition changed will still have an update for the whole table object. Catalogd still sends the whole table thrift object to the catalog topic.

      Note that this is a catalogd-only flag. Don't need to be set on impalads since impalad can process both incremental and non-incremental catalog updates.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stigahuang Quanlong Huang

              Dates

              • Created:
                Updated:

                Issue deployment