Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13699

Configuration does not substitute multiple references to the same var

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: conf
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      This changes the config var cycle detection introduced in 3.0.0-alpha1 by HADOOP-6871 such that it detects single-variable but not multi-variable loops. This also fixes resolution of multiple specifications of the same variable in a config value.
      Show
      This changes the config var cycle detection introduced in 3.0.0-alpha1 by HADOOP-6871 such that it detects single-variable but not multi-variable loops. This also fixes resolution of multiple specifications of the same variable in a config value.

      Description

      Config var loop detection was originally introduced by HADOOP-6871. Due to cycle detection changes in the trunk patch for HADOOP-11506, resolution for multiple references to the same variable no longer resolved, e.g.

      somekey = "${otherkey} ${otherkey}"
      

      This loop detection business is fragile, expensive, and not in branch-2, so let's reduce it.

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Patch attached. This only attempts to catch the simple case of a variable resolving to itself. Support for detecting multiple variable loops is removed.

          Solving this generally is like detecting a loop in a linked list. Since this is performance sensitive code, I think we'd rather not do that.

          Show
          andrew.wang Andrew Wang added a comment - Patch attached. This only attempts to catch the simple case of a variable resolving to itself. Support for detecting multiple variable loops is removed. Solving this generally is like detecting a loop in a linked list. Since this is performance sensitive code, I think we'd rather not do that.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for finding and fixing this Andrew. I'm +1 on the return-fast change, and looking at HADOOP-6871 it's fixing a bug that's not considered in the original implementation. Ideally we should let people from either jira comment, in case they feel differently.

          And man HADOOP-6871 is 4+ years ago. I hope no one has been waiting for that for the past half-decade. Mind linking the jira (I didn't find a replaces option, maybe supercedes?) and comment there, so this is change is more advertised?

          Show
          xiaochen Xiao Chen added a comment - Thanks for finding and fixing this Andrew. I'm +1 on the return-fast change, and looking at HADOOP-6871 it's fixing a bug that's not considered in the original implementation. Ideally we should let people from either jira comment, in case they feel differently. And man HADOOP-6871 is 4+ years ago. I hope no one has been waiting for that for the past half-decade. Mind linking the jira (I didn't find a replaces option, maybe supercedes ?) and comment there, so this is change is more advertised?
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, thanks for reviewing Xiao!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for reviewing Xiao!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10579 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10579/)
          HADOOP-13699. Configuration does not substitute multiple references to (wang: rev 03060075c53a2cecfbf5f60b6fc77afecf64ace5)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10579 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10579/ ) HADOOP-13699 . Configuration does not substitute multiple references to (wang: rev 03060075c53a2cecfbf5f60b6fc77afecf64ace5) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development