[HIVE-24279] Hive CompactorThread fails to connect to metastore if the connectionURL was not configured in hive-site

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: Metastore
• Assignee: Unassigned
• Reporter: Xianyin Xin
• Created: 15/Oct/20 12:18
• Updated: 15/Oct/20 12:53

Description

I got some exception when i configured transaction (the metastore connection info are configured in hivemetastore-site.xml but not in hive-site.xml),

2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an exception in the main loop of compactor initiator, exiting MetaException(message:Unable to connect to transaction database java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not exist.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown Source)
        at com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
        at com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
        at org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646)
        at org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208)
        at org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74)
Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist.

After some debugging, i found the conf that passed to the CompactorThread was:

  public void setConf(Configuration configuration) {
    // TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to call other methods with
    // it. This should be changed to Configuration once everything that this calls that requires
    // HiveConf is moved to the standalone metastore.
    conf = (configuration instanceof HiveConf) ? (HiveConf)configuration :
        new HiveConf(configuration, HiveConf.class);
  }

However, new HiveConf(configuration, HiveConf.class) would not inherit all the configs come from configuration, actually the configuration will be overwrite by the default values those are not nulls:

  private void initialize(Class<?> cls) {
    hiveJar = (new JobConf(cls)).getJar();

    // preserve the original configuration
    origProp = getAllProperties();

    // Overlay the ConfVars. Note that this ignores ConfVars with null values
    addResource(getConfVarInputStream());

    // Overlay hive-site.xml if it exists
    if (hiveSiteURL != null) {
      addResource(hiveSiteURL);
    }

    // if embedded metastore is to be used as per config so far
    // then this is considered like the metastore server case
    String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS);
    // This is hackery, but having hive-common depend on standalone-metastore is really bad
    // because it will pull all of the metastore code into every module.  We need to check that
    // we aren't using the standalone metastore.  If we are, we should treat it the same as a
    // remote metastore situation.
    if (msUri == null || msUri.isEmpty()) {
      msUri = this.get("metastore.thrift.uris");
    }
    LOG.debug("Found metastore URI of " + msUri);
    if(HiveConfUtil.isEmbeddedMetaStore(msUri)){
      setLoadMetastoreConfig(true);
    }

    // load hivemetastore-site.xml if this is metastore and file exists
    if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) {
      addResource(hivemetastoreSiteUrl);
    }

That is, new HiveConf(configuration, HiveConf.class) is merely a new hiveConf but the configs which have null default values will be overwrite by configuration. A hiveConf would not load hivemetastore-site except that it is an embedded metastore. If hive-site doesn't have the db connection info, CompactorThread would connect to the default derby.