Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4970

Difficult to trace "Connection Refused" in AM Proxying

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: webapp
    • Environment:

      Hadoop-2.6.0-CDH5.5.1, linux

      Description

      In generating an HA YARN config (effectively using my own tools for generation), I missed out the multiple specification of yarn.resourcemanager.webapp.address, which produced a "Connection Refused" similar to that in YARN-800 (because of a misunderstanding about what the proxy address was) from the AM webapp.

      It occurs to me, though, in tracing the code, that the behaviour of hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RMHAUtils.java: getRMHAWebappAddresses(YarnConfiguration) should probably mirror the effect of the startup of the resourcemanager itself, such that if the id-suffixed key doesn't exist, you end up using the yarn.resourcemanager.address with the port number replaced by the one taken from yarn.resourcemanager.webapp.address (or its https equivalent). That is, if you had a config like:

      <configuration>
        <property><name>yarn.resourcemanager.ha.enabled</name>
          <value>true</value></property>
        <property><name>yarn.resourcemanager.ha.rm-ids</name>
          <value>hosta,hostb</value></property>
        <property><name>yarn.resourcemanager.webapp.address</name>
          <value>0.0.0.0:8088</value></property>
        <property><name>yarn.resourcemanager.address.hosta</name>
          <value>hosta:8032</value></property>
        <property><name>yarn.resourcemanager.address.hostb</name>
          <value>hostb:8032</value></property>
      </configuration>
      

      you would end up with ("hosta:8088", "hostb:8088") as the List<String> result of the function above, rather than the current empty list result.

      This would certainly give a better principle of least-surprise for operators of yarn (especially those like myself who ended up configuring it wrongly).

      Thoughts? Reasons why this isn't a good idea? (I'm afraid my Java's a bit read-only which is why I haven't suggested a patch)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mbm Matthew Byng-Maddick
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: