Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22160

Job Configuration Properties Unable to Override hive-site.xml Properties in Master Node

    XMLWordPrintableJSON

    Details

      Description

      When running on the master node, these two ways of creating HiveConf can result in different Hive configurations:

      HiveConf hiveConf = HCatUtil.getHiveConf(job.getConfiguration());
      

      the above one referred as Style1, and

      HiveConf hiveConf = new HiveConf();
      hiveConf.addResource(job.getConfiguration());
      

      the above one referred as Style2. If there are properties exist both in hiveSite and jobConf. For example, in hive-site.xml, we have property 

      hive.metastore.uris = AAA

      And if at the same time in jobConf, such as action configurations in Oozie's workflow.xml, we have 

      hive.metastore.uris = BBB

      Then in Style1, the "hive.metastore.uris" property in hiveConf is "AAA" and in Style2, the property in hiveConf is "BBB".

      Here is another example to illustrate this. Suppose in hiveSite, "hive.metastore.uris" is AAA and in jobConf it is BBB, when executing the following code:

      HiveConf hiveConf = HCatUtil.getHiveConf(job.getConfiguration());
      log.info("the hive metastore uri in hiveconf first is: " + hiveConf.get("hive.metastore.uris"));
      hiveConf = HCatUtil.getHiveConf(job.getConfiguration());
      log.info("the hive metastore uri in hiveconf second is: " + hiveConf.get("hive.metastore.uris"));
      

      And the log output is:

      the hive metastore uri in hiveconf first is: AAA
      the hive metastore uri in hiveconf first is: BBB

      In all, the jobConf does not got set to the configuration as expected when there's already such a property existed in hive site. This happen only on the master node but not on task nodes.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zhtpandog Haotian Zhang
                Reporter:
                zhtpandog Haotian Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h