Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: conf
    • Labels:

      Description

      We should allow users to use the more compact form of xml elements. For example, we could allow:

      <property name="mapred.local.dir" value="/disk/0/mapred,/disk/1/mapred"/>
      

      The old format would also be supported.

        Activity

        Hide
        aw Allen Wittenauer added a comment -

        +1 for less <, >, and /.

        Show
        aw Allen Wittenauer added a comment - +1 for less <, >, and /.
        Hide
        sekikn Kengo Seki added a comment -

        Attaching a draft patch.

        Show
        sekikn Kengo Seki added a comment - Attaching a draft patch.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12670973/HADOOP-6964.patch
        against trunk revision ef784a2.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.crypto.random.TestOsSecureRandom

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4800//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/4800//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-common.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4800//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670973/HADOOP-6964.patch against trunk revision ef784a2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.crypto.random.TestOsSecureRandom +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4800//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/4800//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4800//console This message is automatically generated.
        Hide
        aw Allen Wittenauer added a comment -

        Findbugs failure is unrelated. See HADOOP-11122 .

        Show
        aw Allen Wittenauer added a comment - Findbugs failure is unrelated. See HADOOP-11122 .
        Hide
        sekikn Kengo Seki added a comment -

        Thank you Allen Wittenauer. Unit test failure also seems unrelated. See HADOOP-11125.

        Show
        sekikn Kengo Seki added a comment - Thank you Allen Wittenauer . Unit test failure also seems unrelated. See HADOOP-11125 .
        Hide
        sekikn Kengo Seki added a comment -

        Current patch implementation is:

        • Attributes and Elements can be mixed. For example,
        <property name="foo" final="true">
          <value>bar</value>
          <source>baz</source>
        </property>
        

        interpreted as {"name": "foo", "value": "bar", "final": "true", "sources": ["baz"]}.

        • If both attribute and element with the identical name are simultaneously defined, the latter wins. But with regard to source, the attribute is added first, and the elements are appended to the attribute.
        <property name="foo" source="a">
          <name>bar</name>
          <value>baz</value>
          <source>b</source>
          <source>c</source>
        </property>
        

        interpreted as {"name": "bar", "value": "baz", "sources": ["a", "b", "c"]}.

        Is this implementation reasonable? Can anybody review this patch?

        Show
        sekikn Kengo Seki added a comment - Current patch implementation is: Attributes and Elements can be mixed. For example, <property name= "foo" final= "true" > <value> bar </value> <source> baz </source> </property> interpreted as {"name": "foo", "value": "bar", "final": "true", "sources": ["baz"]}. If both attribute and element with the identical name are simultaneously defined, the latter wins. But with regard to source, the attribute is added first, and the elements are appended to the attribute. <property name= "foo" source= "a" > <name> bar </name> <value> baz </value> <source> b </source> <source> c </source> </property> interpreted as {"name": "bar", "value": "baz", "sources": ["a", "b", "c"]}. Is this implementation reasonable? Can anybody review this patch?
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12670973/HADOOP-6964.patch
        against trunk revision 9112f09.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5596//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5596//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670973/HADOOP-6964.patch against trunk revision 9112f09. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5596//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5596//console This message is automatically generated.
        Hide
        ozawa Tsuyoshi Ozawa added a comment -

        Kengo Seki cc: Allen Wittenauer Thank you for the contribution. From operator's view, I think it's useful to validate multiple definition of attributes and elements and abort with RuntimeException at the time since sometimes it's hard to debug the wrong configurations. What do you think?

        Show
        ozawa Tsuyoshi Ozawa added a comment - Kengo Seki cc: Allen Wittenauer Thank you for the contribution. From operator's view, I think it's useful to validate multiple definition of attributes and elements and abort with RuntimeException at the time since sometimes it's hard to debug the wrong configurations. What do you think?
        Hide
        aw Allen Wittenauer added a comment -

        abort with RuntimeException at the time

        This would be disastrous during a 'refresh' or 'renew' or whatever-the-HDFS-team-decides-to-call-it-this-week operation. A running daemon would bite the dust!

        sometimes it's hard to debug the wrong configurations.

        Yup, agree. Luckily, HADOOP-7947 (which Kengo Seki is also working on!) covers the validation part. It isn't at runtime, however.

        Show
        aw Allen Wittenauer added a comment - abort with RuntimeException at the time This would be disastrous during a 'refresh' or 'renew' or whatever-the-HDFS-team-decides-to-call-it-this-week operation. A running daemon would bite the dust! sometimes it's hard to debug the wrong configurations. Yup, agree. Luckily, HADOOP-7947 (which Kengo Seki is also working on!) covers the validation part. It isn't at runtime, however.
        Hide
        aw Allen Wittenauer added a comment -

        Also, I'm +1 barring any further comments from the other folks watching. I'll commit this EOD California time otherwise.

        Show
        aw Allen Wittenauer added a comment - Also, I'm +1 barring any further comments from the other folks watching. I'll commit this EOD California time otherwise.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        validate multiple definition of attributes and elements and abort with RuntimeException

        I'm thinking it would be sufficient to output warning message as follows:

        Configuration.java
                  if ("name".equals(field.getTagName()) && field.hasChildNodes())
                    if (attr != null) {
                      LOG.warn("Detected multiple definitions. Property name: " + attr + 
                               " will be overwritten.");
                    }
                    attr = StringInterner.weakIntern(
                        ((Text)field.getFirstChild()).getData().trim());
        

        However, I'm okay with doing this by a separate jira. +1 for the patch, thanks Kengo and Allen.

        Show
        ajisakaa Akira Ajisaka added a comment - validate multiple definition of attributes and elements and abort with RuntimeException I'm thinking it would be sufficient to output warning message as follows: Configuration.java if ( "name" .equals(field.getTagName()) && field.hasChildNodes()) if (attr != null ) { LOG.warn( "Detected multiple definitions. Property name: " + attr + " will be overwritten." ); } attr = StringInterner.weakIntern( ((Text)field.getFirstChild()).getData().trim()); However, I'm okay with doing this by a separate jira. +1 for the patch, thanks Kengo and Allen.
        Hide
        aw Allen Wittenauer added a comment -

        Actually, wouldn't multiple entries trigger every time someone overwrites a value in the JobConf? That sounds bad.

        Show
        aw Allen Wittenauer added a comment - Actually, wouldn't multiple entries trigger every time someone overwrites a value in the JobConf? That sounds bad.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Actually, wouldn't multiple entries trigger every time someone overwrites a value in the JobConf?

        You are right. Logging is not a good idea.

        Show
        ajisakaa Akira Ajisaka added a comment - Actually, wouldn't multiple entries trigger every time someone overwrites a value in the JobConf? You are right. Logging is not a good idea.
        Hide
        aw Allen Wittenauer added a comment -

        I've committed this to trunk.

        Thanks!

        Show
        aw Allen Wittenauer added a comment - I've committed this to trunk. Thanks!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7034 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7034/)
        HADOOP-6964. Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7034 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7034/ ) HADOOP-6964 . Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/)
        HADOOP-6964. Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/ ) HADOOP-6964 . Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/830/)
        HADOOP-6964. Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/830/ ) HADOOP-6964 . Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/)
        HADOOP-6964. Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/ ) HADOOP-6964 . Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/)
        HADOOP-6964. Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/ ) HADOOP-6964 . Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/)
        HADOOP-6964. Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/ ) HADOOP-6964 . Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2047 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2047/)
        HADOOP-6964. Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2047 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2047/ ) HADOOP-6964 . Allow compact property description in xml (Kengo Seki via aw) (aw: rev af3aadf04f0d6aff19fff99fe18c9b3feae2c529) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        sekikn Kengo Seki added a comment -

        Thank you Allen Wittenauer, Tsuyoshi Ozawa, Akira Ajisaka.
        On HADOOP-7947, I’ll try to make the validator to warn if attributes and elements are duplicated.

        Show
        sekikn Kengo Seki added a comment - Thank you Allen Wittenauer , Tsuyoshi Ozawa , Akira Ajisaka . On HADOOP-7947 , I’ll try to make the validator to warn if attributes and elements are duplicated.

          People

          • Assignee:
            sekikn Kengo Seki
            Reporter:
            owen.omalley Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development