Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9937

Catalogd should send incremental metadata updates in DDL responses to legacy coordinators

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Catalog
    • None

    Description

      After IMPALA-3127, catalogd is able to send incremental metadata updates in partition level. Ideally, legacy coordinators should receive incremental updates in DDL responses. E.g. when adding one partition, only get back the metadata of this partition and some table level updates, not including metadata of other unchanged partitions.

      However, catalogd still sends full table objects in DDL responses because catalogd is not aware of the table metadata cached in coordinator's cache. Catalogd can't compare them to send incremental updates. To support this, the Catalogd RPC requests should include the partition id list of the target table, catalogd uses it to generate incremental metadata updates in responses.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            stigahuang Quanlong Huang

            Dates

              Created:
              Updated:

              Slack

                Issue deployment