Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6910

MapReduceTrackingUriPlugin can not return the right URI of history server with HTTPS

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.3, 2.8.1, 3.0.0-alpha3
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: jobhistoryserver
    • Labels:
      None

      Description

      When the MapReduceTrackingUriPlugin enabled, the URI requests from proxy server or RM UI which are also out of yarn.resourcemanager.max-completed-applications should be redirect to the history server URI.
      But when I access a HTTPS history server with the properties:

      <property>
      <name>mapreduce.jobhistory.http.policy</name>
      <value>HTTPS_ONLY</value>
      </property>
      <property>
      <name>mapreduce.jobhistory.webapp.https.address</name>
      <value>history.example.com:12345</value>
      </property>

      The MapReduceTrackingUriPlugin still returns a default HTTP URI:
      http://0.0.0.0:19888
      or
      http://history.example.com:67890
      if mapreduce.jobhistory.webapp.address is engaged at same time.

      <property>
      <name>mapreduce.jobhistory.webapp.address</name>
      <value>history.example.com:67890</value>
      </property>

      1. MAPREDUCE-6910.001.patch
        5 kB
        Lantao Jin
      2. MAPREDUCE-6910.002.patch
        2 kB
        Lantao Jin

        Activity

        Hide
        cltlfcjin Lantao Jin added a comment -

        The root cause is that the static object httpPolicyInJHS in MRWebAppUtil has no chance to initialize. It always uses the default value null and below methods will return the value like when HTTP

          public static String getJHSWebappScheme() {
            return httpPolicyInJHS == HttpConfig.Policy.HTTPS_ONLY ? "https://" : "http://";
          }
        
          public static String getJHSWebappURLWithoutScheme(Configuration conf) {
            if (httpPolicyInJHS == Policy.HTTPS_ONLY) {
              return conf.get(JHAdminConfig.MR_HISTORY_WEBAPP_HTTPS_ADDRESS,
                  JHAdminConfig.DEFAULT_MR_HISTORY_WEBAPP_HTTPS_ADDRESS);
            } else {
              return conf.get(JHAdminConfig.MR_HISTORY_WEBAPP_ADDRESS,
                  JHAdminConfig.DEFAULT_MR_HISTORY_WEBAPP_ADDRESS);
            }
          }
        

        Attach a patch to fix.

        Show
        cltlfcjin Lantao Jin added a comment - The root cause is that the static object httpPolicyInJHS in MRWebAppUtil has no chance to initialize. It always uses the default value null and below methods will return the value like when HTTP public static String getJHSWebappScheme() { return httpPolicyInJHS == HttpConfig.Policy.HTTPS_ONLY ? "https: //" : "http://" ; } public static String getJHSWebappURLWithoutScheme(Configuration conf) { if (httpPolicyInJHS == Policy.HTTPS_ONLY) { return conf.get(JHAdminConfig.MR_HISTORY_WEBAPP_HTTPS_ADDRESS, JHAdminConfig.DEFAULT_MR_HISTORY_WEBAPP_HTTPS_ADDRESS); } else { return conf.get(JHAdminConfig.MR_HISTORY_WEBAPP_ADDRESS, JHAdminConfig.DEFAULT_MR_HISTORY_WEBAPP_ADDRESS); } } Attach a patch to fix.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 43s Maven dependency ordering for branch
        +1 mvninstall 13m 5s trunk passed
        +1 compile 1m 46s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 0m 55s trunk passed
        +1 javadoc 0m 31s trunk passed
        0 mvndep 0m 6s Maven dependency ordering for patch
        +1 mvninstall 0m 32s the patch passed
        +1 compile 1m 43s the patch passed
        +1 javac 1m 43s the patch passed
        +1 checkstyle 0m 26s the patch passed
        +1 mvnsite 0m 33s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 10s the patch passed
        +1 javadoc 0m 27s the patch passed
        +1 unit 0m 43s hadoop-mapreduce-client-common in the patch passed.
        +1 unit 0m 13s hadoop-mapreduce-client-hs-plugins in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        26m 1s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue MAPREDUCE-6910
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875397/MAPREDUCE-6910.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 83c731df9421 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / fa1aaee
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7015/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7015/console
        Powered by Apache Yetus 0.4.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 13m 5s trunk passed +1 compile 1m 46s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 0m 55s trunk passed +1 javadoc 0m 31s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 0m 32s the patch passed +1 compile 1m 43s the patch passed +1 javac 1m 43s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 10s the patch passed +1 javadoc 0m 27s the patch passed +1 unit 0m 43s hadoop-mapreduce-client-common in the patch passed. +1 unit 0m 13s hadoop-mapreduce-client-hs-plugins in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 26m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue MAPREDUCE-6910 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875397/MAPREDUCE-6910.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 83c731df9421 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fa1aaee Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7015/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7015/console Powered by Apache Yetus 0.4.0 http://yetus.apache.org This message was automatically generated.
        Hide
        cltlfcjin Lantao Jin added a comment -

        Yi Liu and Benoy Antony, could you help to review this?

        Show
        cltlfcjin Lantao Jin added a comment - Yi Liu and Benoy Antony , could you help to review this?
        Hide
        raviprak Ravi Prakash added a comment -

        Hi Lantao! Thank you for your contribution. How did you discover this issue? It seems to me that as long as the server calls MRWebAppUtil.initialize(conf); (which the MR-JHS does), we should be fine, right?

        I think the unit test is incorrect. So when I add the above line

          @Test
          public void testProducesHistoryServerUriWithHTTPS()
              throws URISyntaxException {
            final String historyAddress = "example.net:404040";
            YarnConfiguration conf = new YarnConfiguration();
            conf.set(JHAdminConfig.MR_HS_HTTP_POLICY,
                HttpConfig.Policy.HTTPS_ONLY.name());
            conf.set(JHAdminConfig.MR_HISTORY_WEBAPP_HTTPS_ADDRESS, historyAddress);
            MRWebAppUtil.initialize(conf);
            MapReduceTrackingUriPlugin plugin = new MapReduceTrackingUriPlugin();
            plugin.setConf(conf);
            ApplicationId id = ApplicationId.newInstance(6384623L, 5);
            String jobSuffix = id.toString().replaceFirst("^application_", "job_");
            URI expected =
                new URI("https://" + historyAddress + "/jobhistory/job/" + jobSuffix);
            URI actual = plugin.getTrackingUri(id);
            assertEquals(expected, actual);
          }
        

        the test seems to pass. Am I missing something?

        Show
        raviprak Ravi Prakash added a comment - Hi Lantao! Thank you for your contribution. How did you discover this issue? It seems to me that as long as the server calls MRWebAppUtil.initialize(conf); (which the MR-JHS does), we should be fine, right? I think the unit test is incorrect. So when I add the above line @Test public void testProducesHistoryServerUriWithHTTPS() throws URISyntaxException { final String historyAddress = "example.net:404040" ; YarnConfiguration conf = new YarnConfiguration(); conf.set(JHAdminConfig.MR_HS_HTTP_POLICY, HttpConfig.Policy.HTTPS_ONLY.name()); conf.set(JHAdminConfig.MR_HISTORY_WEBAPP_HTTPS_ADDRESS, historyAddress); MRWebAppUtil.initialize(conf); MapReduceTrackingUriPlugin plugin = new MapReduceTrackingUriPlugin(); plugin.setConf(conf); ApplicationId id = ApplicationId.newInstance(6384623L, 5); String jobSuffix = id.toString().replaceFirst( "^application_" , "job_" ); URI expected = new URI( "https: //" + historyAddress + "/jobhistory/job/" + jobSuffix); URI actual = plugin.getTrackingUri(id); assertEquals(expected, actual); } the test seems to pass. Am I missing something?
        Hide
        cltlfcjin Lantao Jin added a comment -

        Ravi Prakash Thanks your review. Yes, MRWebAppUtil.initialize(conf); can fix it also. But this will import MRWebAppUtil from outer module project. I can offer the new patch which use MRWebAppUtil.initialize to fix this issue.

        Show
        cltlfcjin Lantao Jin added a comment - Ravi Prakash Thanks your review. Yes, MRWebAppUtil.initialize(conf); can fix it also. But this will import MRWebAppUtil from outer module project. I can offer the new patch which use MRWebAppUtil.initialize to fix this issue.
        Hide
        cltlfcjin Lantao Jin added a comment -

        Ravi Prakash Whatever embedded or standalone proxy server deployed, WebAppProxyServlet will always be created. So put MRWebAppUtil.initialize(conf); in WebAppProxyServlet can fix it. It's a very easy patch and I doesn't provide an UT. What do you think?

        Show
        cltlfcjin Lantao Jin added a comment - Ravi Prakash Whatever embedded or standalone proxy server deployed, WebAppProxyServlet will always be created. So put MRWebAppUtil.initialize(conf); in WebAppProxyServlet can fix it. It's a very easy patch and I doesn't provide an UT. What do you think?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 52s trunk passed
        +1 compile 0m 16s trunk passed
        +1 checkstyle 0m 11s trunk passed
        +1 mvnsite 0m 18s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 12s trunk passed
        +1 mvninstall 0m 14s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        +1 checkstyle 0m 8s the patch passed
        +1 mvnsite 0m 15s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 0m 28s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 0m 21s hadoop-yarn-server-web-proxy in the patch passed.
        0 asflicense 0m 7s ASF License check generated no output?
        18m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue MAPREDUCE-6910
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877062/MAPREDUCE-6910.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
        uname Linux 8f28e44d85a5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / cf0d084
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7027/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7027/console
        Powered by Apache Yetus 0.4.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 52s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 11s trunk passed +1 mvnsite 0m 18s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 14s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 checkstyle 0m 8s the patch passed +1 mvnsite 0m 15s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 0m 21s hadoop-yarn-server-web-proxy in the patch passed. 0 asflicense 0m 7s ASF License check generated no output? 18m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue MAPREDUCE-6910 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877062/MAPREDUCE-6910.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 8f28e44d85a5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cf0d084 Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7027/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7027/console Powered by Apache Yetus 0.4.0 http://yetus.apache.org This message was automatically generated.
        Hide
        raviprak Ravi Prakash added a comment -

        Hi Lantao! You are right that Yarn should be oblivious of Application details (be it MapReduce or Tez or Spark or some other application). Thank you for pointing that out. I think your first patch makes sense.

        +1 for MAPREDUCE-6910.001.patch . LGTM. I'll commit it EOD unless there are objections.

        Show
        raviprak Ravi Prakash added a comment - Hi Lantao! You are right that Yarn should be oblivious of Application details (be it MapReduce or Tez or Spark or some other application). Thank you for pointing that out. I think your first patch makes sense. +1 for MAPREDUCE-6910 .001.patch . LGTM. I'll commit it EOD unless there are objections.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12004 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12004/)
        MAPREDUCE-6910. MapReduceTrackingUriPlugin can not return the right URI (raviprak: rev 43f0503286eccbc6bb8ae77584b635bfd0c48e50)

        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRWebAppUtil.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestMapReduceTrackingUriPlugin.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12004 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12004/ ) MAPREDUCE-6910 . MapReduceTrackingUriPlugin can not return the right URI (raviprak: rev 43f0503286eccbc6bb8ae77584b635bfd0c48e50) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRWebAppUtil.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestMapReduceTrackingUriPlugin.java
        Hide
        cltlfcjin Lantao Jin added a comment -

        Thanks Ravi Prakash. Do I need to delete the 002 attachment after resolved? (Could you help to add me to the assignee?)

        Show
        cltlfcjin Lantao Jin added a comment - Thanks Ravi Prakash . Do I need to delete the 002 attachment after resolved? (Could you help to add me to the assignee?)
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks Lantao for your contribution. I look forward to more.

        Do I need to delete the 002 attachment after resolved?

        No. That's ok.

        (Could you help to add me to the assignee?)

        Please let me work on that. I'll try to do that soon.

        Show
        raviprak Ravi Prakash added a comment - Thanks Lantao for your contribution. I look forward to more. Do I need to delete the 002 attachment after resolved? No. That's ok. (Could you help to add me to the assignee?) Please let me work on that. I'll try to do that soon.
        Hide
        raviprak Ravi Prakash added a comment -

        I've committed this to branch trunk, branch-2, and branch-2.8.

        Show
        raviprak Ravi Prakash added a comment - I've committed this to branch trunk, branch-2, and branch-2.8.
        Hide
        raviprak Ravi Prakash added a comment -

        After having committed this, I realize that the mapreduce configurations should be in mapred-site.xml . The YARN ResourceManager is not going to load that file. I'm going to re-open this JIRA and we can brainstorm some ideas to fix this.

        Show
        raviprak Ravi Prakash added a comment - After having committed this, I realize that the mapreduce configurations should be in mapred-site.xml . The YARN ResourceManager is not going to load that file. I'm going to re-open this JIRA and we can brainstorm some ideas to fix this.
        Hide
        raviprak Ravi Prakash added a comment -

        The reason I am not reverting the patch for now, is that this atleast gives a work-around (undesirable as it is, bleeding mapred configuration in yarn-site.xml).

        Show
        raviprak Ravi Prakash added a comment - The reason I am not reverting the patch for now, is that this atleast gives a work-around (undesirable as it is, bleeding mapred configuration in yarn-site.xml).
        Hide
        cltlfcjin Lantao Jin added a comment -

        Ravi Prakash, ResourceManager can load the mapred-site.xml. No need to write the mapreduce. properties to yarn-site.xml.
        Here is the mapred-site.xml loading path: (I hack the code and throw a runtime exception to print the stack trace)

        2017-07-14 04:16:19,029 WARN org.apache.hadoop.mapreduce.util.ConfigUtil: Load mapred-site.xml from
        java.lang.RuntimeException: Load mapred-site.xml from
                at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:48)
                at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:119)
                at java.lang.Class.forName0(Native Method)
                at java.lang.Class.forName(Class.java:348)
                at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
                at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:94)
                at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
                at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
                at org.apache.hadoop.yarn.conf.ConfigurationProviderFactory.getConfigurationProvider(ConfigurationProviderFactory.java:52)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:205)
                at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1212)
        
        Show
        cltlfcjin Lantao Jin added a comment - Ravi Prakash , ResourceManager can load the mapred-site.xml. No need to write the mapreduce. properties to yarn-site.xml. Here is the mapred-site.xml loading path: (I hack the code and throw a runtime exception to print the stack trace) 2017-07-14 04:16:19,029 WARN org.apache.hadoop.mapreduce.util.ConfigUtil: Load mapred-site.xml from java.lang.RuntimeException: Load mapred-site.xml from at org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:48) at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:119) at java.lang. Class .forName0(Native Method) at java.lang. Class .forName( Class .java:348) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:94) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.yarn.conf.ConfigurationProviderFactory.getConfigurationProvider(ConfigurationProviderFactory.java:52) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:205) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1212)
        Hide
        cltlfcjin Lantao Jin added a comment - - edited

        The code I test is based on v2.7.1.
        From the stack trace, we can find it will set JobConf when reflection.

        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:94)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)

          public static void setConf(Object theObject, Configuration conf) {
            if (conf != null) {
              if (theObject instanceof Configurable) {
                ((Configurable) theObject).setConf(conf);
              }
              setJobConf(theObject, conf);
            }
          }
        

        Then the mapred-site.xml will be loaded at last.

        public class JobConf extends Configuration {
          static{
            ConfigUtil.loadResources();
          }
        
        public class ConfigUtil {
          /**
           * Adds all the deprecated keys. Loads mapred-default.xml and mapred-site.xml
           */
          public static void loadResources() {
            addDeprecatedKeys();
            Configuration.addDefaultResource("mapred-default.xml");
            Configuration.addDefaultResource("mapred-site.xml");
            Configuration.addDefaultResource("yarn-default.xml");
            Configuration.addDefaultResource("yarn-site.xml");
          }
        
        Show
        cltlfcjin Lantao Jin added a comment - - edited The code I test is based on v2.7.1. From the stack trace, we can find it will set JobConf when reflection. at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:94) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) public static void setConf( Object theObject, Configuration conf) { if (conf != null ) { if (theObject instanceof Configurable) { ((Configurable) theObject).setConf(conf); } setJobConf(theObject, conf); } } Then the mapred-site.xml will be loaded at last. public class JobConf extends Configuration { static { ConfigUtil.loadResources(); } public class ConfigUtil { /** * Adds all the deprecated keys. Loads mapred- default .xml and mapred-site.xml */ public static void loadResources() { addDeprecatedKeys(); Configuration.addDefaultResource( "mapred- default .xml" ); Configuration.addDefaultResource( "mapred-site.xml" ); Configuration.addDefaultResource( "yarn- default .xml" ); Configuration.addDefaultResource( "yarn-site.xml" ); }
        Hide
        cltlfcjin Lantao Jin added a comment -

        The patch is aimed at fixing the uninitialised variable to avoid always using the default HTTP in HTTPS case. And it works for this purpose. J

        Show
        cltlfcjin Lantao Jin added a comment - The patch is aimed at fixing the uninitialised variable to avoid always using the default HTTP in HTTPS case. And it works for this purpose. J
        Hide
        raviprak Ravi Prakash added a comment -

        I guess one could argue that for the MapReduceTrackingUriPlugin to work, the ResourceManager must have mapred-site.xml loaded . Thanks for the clarification Lantao. I've assigned the issue to you and will close it.

        Show
        raviprak Ravi Prakash added a comment - I guess one could argue that for the MapReduceTrackingUriPlugin to work, the ResourceManager must have mapred-site.xml loaded . Thanks for the clarification Lantao. I've assigned the issue to you and will close it.

          People

          • Assignee:
            cltlfcjin Lantao Jin
            Reporter:
            cltlfcjin Lantao Jin
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development