[HIVE-10381] Allow pushing of property-key-value based predicate filter to Metastore dropPartitions

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Import/Export
• Assignee: Unassigned Assign to me
• Reporter: Sushanth Sowmyan
• Created: 17/Apr/15 18:33
• Updated: 17/Apr/15 18:33

Description

After HIVE-10228, we now have a case wherein we assume, based on our knowledge of how replication will work, that DROP_PARTITION replication will act on a per-partition level.

In order to be robust, however, we do handle the case where this might not be the case, wherein the DROP_PARTITION replication command generated would work across multiple partitions, it's just that that handling is very inefficient if ever used, since it has to fetch each of those partitions from the metastore, decide whether or not it matches the ReplicationSpec.allowEventReplacementInto filter, and then drop it.

Ideally, we should allow pushing this filter to the metastore, and let the metastore do a smart drop based on that.

    ReplicationSpec replicationSpec = dropTbl.getReplicationSpec();
    if (replicationSpec.isInReplicationScope()){
      // TODO: Current implementation of replication will result in DROP_PARTITION under replication
      // scope being called per-partition instead of multiple partitions. However, to be robust, we
      // must still handle the case of multiple partitions in case this assumption changes in the
      // future. However, if this assumption changes, we will not be very performant if we fetch
      // each partition one-by-one, and then decide on inspection whether or not this is a candidate
      // for dropping. Thus, we need a way to push this filter (replicationSpec.allowEventReplacementInto)
      // to the  metastore to allow it to do drop a partition or not, depending on a Predicate on the
      // parameter key values.
      for (DropTableDesc.PartSpec partSpec : dropTbl.getPartSpecs()){
        try {
          for (Partition p : Iterables.filter(
              db.getPartitionsByFilter(tbl, partSpec.getPartSpec().getExprString()),
              replicationSpec.allowEventReplacementInto())){
            db.dropPartition(tbl.getDbName(),tbl.getTableName(),p.getValues(),true);
          }
        } catch (NoSuchObjectException e){
          // ignore NSOE because that means there's nothing to drop.
        } catch (Exception e) {
          throw new HiveException(e.getMessage(), e);
        }
      }
      return;
    }