Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14701

Configuration can log misleading warnings about an attempt to override final parameter

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: conf
    • Labels:
      None

      Description

      The class Configuration (in hadoop-common-project) is used to hold configuration properties.

      The method loadResources() is used to load properties from a list of default resources into a parameter called “properties” (which happens to always be the field called “properties”).

      This is done by calling the method loadResource() which is used to load properties from a single resource into a parameter called “properties” (which happens to always be the field called “properties”).

      This is done by calling loadProperty() which is used to load a single property into a parameter called “properties” (which is sometimes, but not always, the field called “properties”).

      In loadProperty() there is some logic for dealing with properties that are marked as final.
      (1) a final property is not set into the “properties” parameter.
      (2) a warning is printed for a final property if the value of the property with the same name in the “properties” parameter is not equal to the new value, and the property name is in the finalParameters field.

      When loadResource() is reading properties from a stream it creates a temporary Properties object that it populates by calling loadProperty(). This is the case where loadProperty() has a parameter called “properties” which is not the same as the field called “properties”).
      When loadProperty() is called on a final property then because of (1) it is not set into the “properties” parameter. But sometimes the property name is already in the finalParameters field and now (2) the warning is triggered because the property is not present in the “properties” parameter (although it IS present in the “properties” field). The warning is misleading and confusing.

      Possible Fix

      One way of looking at the problem is that loadProperty is using the finalParameters field in combination with the “properties” parameter. Change the code to only generate a warning for a final property if the value of the property with the same name in the “properties” field is not equal to the new value, and the property name is in the finalParameters field.

      I’ve made a patch containing this tiny change.

      Note that I have done very minimal testing of this fix

      Example warning:

      2017-07-25 12:14:24,639 WARN org.apache.hadoop.conf.Configuration: [HiveServer2-Background-Pool: Thread-77]: hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.

        Attachments

        1. HADOOP-14701.4.patch
          10 kB
          Andrew Sherman
        2. HADOOP-14701.3.patch
          6 kB
          Andrew Sherman
        3. HADOOP-14701.2.patch
          6 kB
          Andrew Sherman
        4. HADOOP-14701.1.patch
          1 kB
          Andrew Sherman

          Activity

            People

            • Assignee:
              asherman Andrew Sherman
              Reporter:
              asherman Andrew Sherman
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: