Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-149 [Umbrella] ResourceManager (RM) Fail-over
  3. YARN-3711

Documentation of ResourceManager HA should explain configurations about listen addresses

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There should be explanation about webapp address in addition to RPC address.

      AM proxy filter needs explicit definition of yarn.resourcemanager.webapp.address.rm-id and/or yarn.resourcemanager.webapp.https.address.rm-id to get proper addresses in RM-HA mode.

      1. YARN-3711.002.patch
        2 kB
        Masatake Iwasaki
      2. YARN-3711.003.patch
        3 kB
        Masatake Iwasaki

        Issue Links

          Activity

          Hide
          iwasakims Masatake Iwasaki added a comment -

          I attached patch. 002 fixes markdown formatting nits too.

          Show
          iwasakims Masatake Iwasaki added a comment - I attached patch. 002 fixes markdown formatting nits too.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 54s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 56s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 14s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735274/YARN-3711.002.patch
          Optional Tests site
          git revision trunk / 56996a6
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8081/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 56s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 14s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735274/YARN-3711.002.patch Optional Tests site git revision trunk / 56996a6 Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8081/console This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Though HAUtil.verifyAndSetConfiguration updates the configuration for HA on RM node, it only cares the rm-id of the node. AMs running on slave nodes need to know webapp addresses of all RMs. The URL of application shown in RM UI refers to wrong host name without explicit definition of webapp addresses.

          Show
          iwasakims Masatake Iwasaki added a comment - Though HAUtil.verifyAndSetConfiguration updates the configuration for HA on RM node, it only cares the rm-id of the node. AMs running on slave nodes need to know webapp addresses of all RMs. The URL of application shown in RM UI refers to wrong host name without explicit definition of webapp addresses.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Masatake Iwasaki thank you for updating.

          +| `yarn.resourcemanager.webapp.https.address.`rm-id | For each rm-id, specify https address of the RM web application corresponds to. |
          +| `yarn.resourcemanager.webapp.address.`rm-id | For each rm-id, specify http address of the RM web application corresponds to. |

          Should we should describe that y.r.webapp.https.address is needed only when yarn.http.policy is HTTPS_ONLY, while y.r.webapp.address is needed only when yarn.http.policy is HTTP_ONLY?

            @Private
            public static List<String> getServiceAddressConfKeys(Configuration conf) {
              return useHttps(conf) ? RM_SERVICES_ADDRESS_CONF_KEYS_HTTPS
                  : RM_SERVICES_ADDRESS_CONF_KEYS_HTTP;
            }
          
          Show
          ozawa Tsuyoshi Ozawa added a comment - Masatake Iwasaki thank you for updating. +| `yarn.resourcemanager.webapp.https.address.` rm-id | For each rm-id , specify https address of the RM web application corresponds to. | +| `yarn.resourcemanager.webapp.address.` rm-id | For each rm-id , specify http address of the RM web application corresponds to. | Should we should describe that y.r.webapp.https.address is needed only when yarn.http.policy is HTTPS_ONLY, while y.r.webapp.address is needed only when yarn.http.policy is HTTP_ONLY? @Private public static List< String > getServiceAddressConfKeys(Configuration conf) { return useHttps(conf) ? RM_SERVICES_ADDRESS_CONF_KEYS_HTTPS : RM_SERVICES_ADDRESS_CONF_KEYS_HTTP; }
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Sure. I updated the patch. I also added the explanation about yarn.resourcemanager.address, yarn.resourcemanager.scheduler.address, yarn.resourcemanager.resource-tracker.address and yarn.resourcemanager.admin.address in 003.

          Show
          iwasakims Masatake Iwasaki added a comment - Sure. I updated the patch. I also added the explanation about yarn.resourcemanager.address, yarn.resourcemanager.scheduler.address, yarn.resourcemanager.resource-tracker.address and yarn.resourcemanager.admin.address in 003.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 53s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 58s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 14s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739494/YARN-3711.003.patch
          Optional Tests site
          git revision trunk / b8341f1
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8246/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 site 2m 58s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 14s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739494/YARN-3711.003.patch Optional Tests site git revision trunk / b8341f1 Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8246/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          +1, commtting this shortly.

          Show
          ozawa Tsuyoshi Ozawa added a comment - +1, commtting this shortly.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8023 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8023/)
          YARN-3711. Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8023 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8023/ ) YARN-3711 . Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md hadoop-yarn-project/CHANGES.txt
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Committed this to trunk, branch-2, and branch-2.7. Thanks Masatake Iwasaki for your contribution.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Committed this to trunk, branch-2, and branch-2.7. Thanks Masatake Iwasaki for your contribution.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Thanks, Tsuyoshi Ozawa!

          Show
          iwasakims Masatake Iwasaki added a comment - Thanks, Tsuyoshi Ozawa !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #960 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/960/)
          YARN-3711. Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #960 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/960/ ) YARN-3711 . Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/230/)
          YARN-3711. Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/230/ ) YARN-3711 . Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2158/)
          YARN-3711. Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2158/ ) YARN-3711 . Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/219/)
          YARN-3711. Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/219/ ) YARN-3711 . Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/228/)
          YARN-3711. Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/228/ ) YARN-3711 . Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2176/)
          YARN-3711. Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2176/ ) YARN-3711 . Documentation of ResourceManager HA should explain configurations about listen addresses. Contributed by Masatake Iwasaki. (ozawa: rev e8c514373f2d258663497a33ffb3b231d0743b57) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerHA.md hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              iwasakims Masatake Iwasaki
              Reporter:
              iwasakims Masatake Iwasaki
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development