Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14213

Move Configuration runtime check for hadoop-site.xml to initialization

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Move the check for hadoop-site.xml to static initialization of the Configuration class.

      Description

      Each Configuration object that loads defaults checks for hadoop-site.xml. It has been long deprecated and is not present in most if not nearly all installations. The getResource check for hadoop-site.xml has to check the entire classpath since it is not found. This jira proposes to 1) either remove hadoop-site.xml as a default resource or 2) move the check to static initialization of the class so the performance hit is only taken once.

      1. HADOOP-14213.1.patch
        2 kB
        Jonathan Eagles
      2. HADOOP-14213.2.patch
        2 kB
        Jonathan Eagles

        Activity

        Hide
        aw Allen Wittenauer added a comment -

        It has been long deprecated and is not present in most if not nearly all installations.

        We just need to be cognizant of the fact that...

        remove hadoop-site.xml as a default resource

        ... is an incompatible change which couldn't go into branch-2.

        (FWIW, I'll raise my hand as one of the folks who still uses hadoop-site.xml on occasion for non-prod installs since it's easier than dealing with the complete mess that happened after the project split.)

        Show
        aw Allen Wittenauer added a comment - It has been long deprecated and is not present in most if not nearly all installations. We just need to be cognizant of the fact that... remove hadoop-site.xml as a default resource ... is an incompatible change which couldn't go into branch-2. (FWIW, I'll raise my hand as one of the folks who still uses hadoop-site.xml on occasion for non-prod installs since it's easier than dealing with the complete mess that happened after the project split.)
        Hide
        jeagles Jonathan Eagles added a comment -

        This patch moves the decision of using hadoop-site.xml as a default resource to the static initialization block in Configuration. This patch maintains the previous precedence ordering of the core-default.xml, core-site.xml, and hadoop-site.xml.

        Show
        jeagles Jonathan Eagles added a comment - This patch moves the decision of using hadoop-site.xml as a default resource to the static initialization block in Configuration. This patch maintains the previous precedence ordering of the core-default.xml, core-site.xml, and hadoop-site.xml.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        Static init makes sense, currently there's one per instance, which adds up

        Show
        stevel@apache.org Steve Loughran added a comment - Static init makes sense, currently there's one per instance, which adds up
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 14m 35s trunk passed
        +1 compile 20m 48s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        +1 findbugs 1m 28s trunk passed
        +1 javadoc 0m 48s trunk passed
        +1 mvninstall 0m 36s the patch passed
        +1 compile 16m 50s the patch passed
        +1 javac 16m 50s the patch passed
        +1 checkstyle 0m 38s the patch passed
        +1 mvnsite 1m 0s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 32s the patch passed
        +1 javadoc 0m 51s the patch passed
        -1 unit 8m 23s hadoop-common in the patch failed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        72m 42s



        Reason Tests
        Failed junit tests hadoop.ha.TestZKFailoverController



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-14213
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859971/HADOOP-14213.1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 111c6b342c77 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f462e1f
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11880/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11880/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11880/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 35s trunk passed +1 compile 20m 48s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 28s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 16m 50s the patch passed +1 javac 16m 50s the patch passed +1 checkstyle 0m 38s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 51s the patch passed -1 unit 8m 23s hadoop-common in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 72m 42s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14213 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859971/HADOOP-14213.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 111c6b342c77 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f462e1f Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11880/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11880/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11880/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jeagles Jonathan Eagles added a comment -

        The test failure in TestZKFailoverController was intermittent and not related to this patch.

        Show
        jeagles Jonathan Eagles added a comment - The test failure in TestZKFailoverController was intermittent and not related to this patch.
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks Jeagles!

        1. typo : "hadoop-site.xml.xml" -> "hadoop-site.xml" ??
        2. I'm sorry! Could you please elaborate how core-site would take precedence over hadoop-site still? It seems like hadoop-site would override the attributes in core-site, right?
        Show
        raviprak Ravi Prakash added a comment - Thanks Jeagles! typo : "hadoop-site.xml.xml" -> "hadoop-site.xml" ?? I'm sorry! Could you please elaborate how core-site would take precedence over hadoop-site still? It seems like hadoop-site would override the attributes in core-site, right?
        Hide
        jeagles Jonathan Eagles added a comment -

        Wow. Good, catch. Thanks, Ravi Prakash.

        Show
        jeagles Jonathan Eagles added a comment - Wow. Good, catch. Thanks, Ravi Prakash .
        Hide
        jeagles Jonathan Eagles added a comment -

        To elaborate on the precedence, currently the final parameters of core-default.xml and core-site.xml take precedence over hadoop-site.xml and this patch preserves that ordering. It is correct that the hadoop-site.xml configs will override for non-final parameters. That behavior is preserved as well.

        Show
        jeagles Jonathan Eagles added a comment - To elaborate on the precedence, currently the final parameters of core-default.xml and core-site.xml take precedence over hadoop-site.xml and this patch preserves that ordering. It is correct that the hadoop-site.xml configs will override for non-final parameters. That behavior is preserved as well.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 5s trunk passed
        +1 compile 21m 33s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 1m 2s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        +1 findbugs 1m 37s trunk passed
        +1 javadoc 0m 53s trunk passed
        +1 mvninstall 0m 43s the patch passed
        +1 compile 17m 21s the patch passed
        +1 javac 17m 21s the patch passed
        +1 checkstyle 0m 38s the patch passed
        +1 mvnsite 1m 0s the patch passed
        +1 mvneclipse 0m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 34s the patch passed
        +1 javadoc 0m 49s the patch passed
        -1 unit 8m 1s hadoop-common in the patch failed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        72m 22s



        Reason Tests
        Failed junit tests hadoop.security.TestKDiag
          hadoop.net.TestDNS



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-14213
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860079/HADOOP-14213.2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8cd44d8121f8 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 59d6925
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11892/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11892/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11892/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 5s trunk passed +1 compile 21m 33s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 43s the patch passed +1 compile 17m 21s the patch passed +1 javac 17m 21s the patch passed +1 checkstyle 0m 38s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 8m 1s hadoop-common in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 72m 22s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14213 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860079/HADOOP-14213.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8cd44d8121f8 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 59d6925 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11892/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11892/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11892/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jeagles Jonathan Eagles added a comment -

        Both TestKDiag and TestDNS are intermittent failures not related to this patch. The overall improvement to Configuration loading on my box is around 2.5% which is very tiny by itself, but a few changes like this will make a big difference.
        What else is needed to get this patch in?

        Show
        jeagles Jonathan Eagles added a comment - Both TestKDiag and TestDNS are intermittent failures not related to this patch. The overall improvement to Configuration loading on my box is around 2.5% which is very tiny by itself, but a few changes like this will make a big difference. What else is needed to get this patch in?
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks for the clarification Jon! LGTM. +1.

        Since you have not removed hadoop-site from default resources, my understanding is that this is not a backward incompatible change.

        Will commit shortly to trunk and branch-2.

        Show
        raviprak Ravi Prakash added a comment - Thanks for the clarification Jon! LGTM. +1. Since you have not removed hadoop-site from default resources, my understanding is that this is not a backward incompatible change. Will commit shortly to trunk and branch-2.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11446 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11446/)
        HADOOP-14213. Move Configuration runtime check for hadoop-site.xml to (raviprak: rev 595f62e362c08704d6fb692e21c97b512bc7ec49)

        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11446 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11446/ ) HADOOP-14213 . Move Configuration runtime check for hadoop-site.xml to (raviprak: rev 595f62e362c08704d6fb692e21c97b512bc7ec49) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java

          People

          • Assignee:
            jeagles Jonathan Eagles
            Reporter:
            jeagles Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development