[FLINK-21829] create HiveCatalog with custom hadoopconfdir first

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: hehuiyuan
• Reporter: hehuiyuan
• Created: 16/Mar/21 15:29
• Updated: 28/Aug/21 11:14
• Resolved: 04/Apr/21 13:48

Description

here is no prompt when the the path to hadoop conf configured is wrong unintentional.

private static HiveConf createHiveConf(
        @Nullable String hiveConfDir, @Nullable String hadoopConfDir) {
    // create HiveConf from hadoop configuration with hadoop conf directory configured.
    Configuration hadoopConf = null;
    if (isNullOrWhitespaceOnly(hadoopConfDir)) {
        for (String possibleHadoopConfPath :
                HadoopUtils.possibleHadoopConfPaths(
                        new org.apache.flink.configuration.Configuration())) {
            hadoopConf = getHadoopConfiguration(possibleHadoopConfPath);
            if (hadoopConf != null) {
                break;
            }
        }
    } else {
        hadoopConf = getHadoopConfiguration(hadoopConfDir);
    }
    if (hadoopConf == null) {
        hadoopConf = new Configuration();
    }
    HiveConf hiveConf = new HiveConf(hadoopConf, HiveConf.class);

It is better to load hadoop conf from possiable hadoop path when the path is wrong.

(1) try to load from the custom hadoop conf path
(2) try to load from possiable hadoop conf path if Configuration hadoopConf is null.
(3) new Configuration if Configuration hadoopConf is null

private static HiveConf createHiveConf(
        @Nullable String hiveConfDir, @Nullable String hadoopConfDir) {
    // create HiveConf from hadoop configuration with hadoop conf directory configured.
    Configuration hadoopConf = null;
    if (!isNullOrWhitespaceOnly(hadoopConfDir)) {
        hadoopConf = getHadoopConfiguration(hadoopConfDir);
    }
    if (hadoopConf == null) {
        for (String possibleHadoopConfPath :
                HadoopUtils.possibleHadoopConfPaths(
                        new org.apache.flink.configuration.Configuration())) {
            hadoopConf = getHadoopConfiguration(possibleHadoopConfPath);
            if (hadoopConf != null) {
                break;
            }
        }
    }
    if (hadoopConf == null) {
        hadoopConf = new Configuration();
    }
    HiveConf hiveConf = new HiveConf(hadoopConf, HiveConf.class);

Issue Links

• links to: GitHub Pull Request #15244
• links to: GitHub Pull Request #15274
• links to: GitHub Pull Request #15427