[HIVE-21818] CBO: Copying TableRelOptHiveTable has metastore traffic

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: CBO
• Labels: pull-request-available
• Assignee: Jesús Camacho Rodríguez
• Reporter: Gopal Vijayaraghavan
• Created: 01/Jun/19 01:08
• Updated: 27/Feb/24 22:24
• Resolved: 07/Jun/19 18:20

Description

While the optimizer is running, whenever it makes a copy of the TableScan to perform potential rewrites, there is Hive metastore traffic from inside CBO optimizer.

  public RelOptHiveTable(RelOptSchema calciteSchema, RelDataTypeFactory typeFactory, List<String> qualifiedTblName,
      RelDataType rowType, Table hiveTblMetadata, List<ColumnInfo> hiveNonPartitionCols,
      List<ColumnInfo> hivePartitionCols, List<VirtualColumn> hiveVirtualCols, HiveConf hconf,
      Map<String, PrunedPartitionList> partitionCache, Map<String, ColumnStatsList> colStatsCache,
      AtomicInteger noColsMissingStats) { ....


    Pair<List<ImmutableBitSet>, List<ImmutableBitSet>> constraintKeys = generateKeys();
    this.keys = constraintKeys.left;
    this.nonNullablekeys = constraintKeys.right;
    this.referentialConstraints = generateReferentialConstraints();
}

This is triggered from the rules as the partition pruner

  protected void perform(RelOptRuleCall call, Filter filter,
      HiveTableScan tScan) {
    // Original table
    RelOptHiveTable hiveTable = (RelOptHiveTable) tScan.getTable();

    // Copy original table scan and table
    HiveTableScan tScanCopy = tScan.copyIncludingTable(tScan.getRowType());
    RelOptHiveTable hiveTableCopy = (RelOptHiveTable) tScanCopy.getTable();
...

    if (StringUtils.equals(hiveTableCopy.getPartitionListKey(), hiveTable.getPartitionListKey())) {
      // Nothing changed, we do not need to produce a new expression
      return;
    }

Issue Links

• links to: GitHub Pull Request #663