Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4524

In CatalogOpExecutor.alterTableAddPartitions() make RPC calls to HMS in batches

    XMLWordPrintableJSON

    Details

      Description

      CatalogOpExecutor.alterTableAddPartitions() should adhere to MAX_PARTITION_UPDATES_PER_RPC limit when making RPC calls to the HMS. We've seen problems in the past when updating a lot of metadata in one RPC to the HMS.

      Note that getPartitionsFromHms() and alterTableCachePartitions() methods called from alterTableAddPartitions() are also affected.

      Currently it is enforced during analysis (in AlterTableAddPartitionStmt.analyze()) that one ALTER TABLE ADD PARTITION statement cannot add more than MAX_PARTITION_UPDATES_PER_RPC partitions. This check can be removed after fixing the RPC calls.

        Attachments

          Activity

            People

            • Assignee:
              philip Philip Martin
              Reporter:
              attilaj Attila Jeges
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: