Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7717

Partition id does not exist exception - Catalog V2

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • Impala 3.1.0
    • Catalog
    • None
    • ghx-label-2

    Description

      Concurrent invalidates with partial RPC on partitioned tables can throw this exception.

      I1016 15:49:03.438048 30197 jni-util.cc:256] java.lang.IllegalArgumentException: Partition id 162 does not exist
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119)
        at org.apache.impala.catalog.HdfsTable.getPartialInfo(HdfsTable.java:1711)
        at org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:2202)
        at org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:2141)
        at org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:237)
      I1016 15:49:03.440939 30197 status.cc:129] IllegalArgumentException: Partition id 162 does not exist
      
       @Override
        public TGetPartialCatalogObjectResponse getPartialInfo(
            TGetPartialCatalogObjectRequest req) throws TableLoadingException {
          
          if (partIds != null) {
            resp.table_info.partitions = Lists.newArrayListWithCapacity(partIds.size());
            for (long partId : partIds) {
              HdfsPartition part = partitionMap_.get(partId);
              Preconditions.checkArgument(part != null, "Partition id %s does not exist",
                  partId); <====
      

      The issue is that the invalidate command can reset the partition IDs and the RPCs could look up with older IDs.

      We should wrap this into an inconsistent metadata fetch exception and retry rather than throwing a RTE.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bharathv Bharath Vissapragada
            bharathv Bharath Vissapragada
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment