Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2553

namenode ports are not configured for non-HA mode

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: deployment
    • Labels:
      None

      Description

      The namenode port, http_port, and https_port are set inside the "HA-mode" condition of the hdfs-site.xml template. They are ignored in non-HA mode.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe closed the pull request at:

          https://github.com/apache/bigtop/pull/151

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe closed the pull request at: https://github.com/apache/bigtop/pull/151
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on the issue:

          https://github.com/apache/bigtop/pull/151

          Pushed upstream with:

          https://github.com/apache/bigtop/commit/cd55c144955e36188640c199b3100fefb3b28612

          I forgot to add a "closes #151" to the commit, so i'm closing this out the old fashioned way.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on the issue: https://github.com/apache/bigtop/pull/151 Pushed upstream with: https://github.com/apache/bigtop/commit/cd55c144955e36188640c199b3100fefb3b28612 I forgot to add a "closes #151" to the commit, so i'm closing this out the old fashioned way.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on the issue:

          https://github.com/apache/bigtop/pull/151

          Oh, of course you're correct. My mistake.

          +1

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on the issue: https://github.com/apache/bigtop/pull/151 Oh, of course you're correct. My mistake. +1
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on the issue:

          https://github.com/apache/bigtop/pull/151

          I think you're saying the `namenode_hosts[host, index]` loop is the only difference, but that's a pretty big difference. The hdfs-site.xml property keys are not the same in HA and non-HA cases.

          For example, property key in HA:
          `dfs.namenode.rpc-address.<%= @nameservice_id %>.nn<%= idx+1 %>`

          Same property key for non-HA:
          `dfs.namenode.rpc-address`

          By the time we check for a single item list and have an `if/else` block rendering different properties based on that list size, I think we're no better than we are now.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on the issue: https://github.com/apache/bigtop/pull/151 I think you're saying the `namenode_hosts [host, index] ` loop is the only difference, but that's a pretty big difference. The hdfs-site.xml property keys are not the same in HA and non-HA cases. For example, property key in HA: `dfs.namenode.rpc-address.<%= @nameservice_id %>.nn<%= idx+1 %>` Same property key for non-HA: `dfs.namenode.rpc-address` By the time we check for a single item list and have an `if/else` block rendering different properties based on that list size, I think we're no better than we are now.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on the issue:

          https://github.com/apache/bigtop/pull/151

          @kwmonroe My point was that, since `namenode_hosts` only contains one item in the non-HA case, and the only difference in the code between the HA case and non-HA case was whether we looped over the code, it could be factored out into a single loop outside of the HA / non-HA check and it would do the right thing by dint of there being only a single item in the list.

          It would save some code repetition, and avoid needing to change the `elif` condition, though the other part of my comment is that that seems strange to me in the first place (but, since it's outside the scope of this change, anyway, avoiding changing it at all seems better).

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on the issue: https://github.com/apache/bigtop/pull/151 @kwmonroe My point was that, since `namenode_hosts` only contains one item in the non-HA case, and the only difference in the code between the HA case and non-HA case was whether we looped over the code, it could be factored out into a single loop outside of the HA / non-HA check and it would do the right thing by dint of there being only a single item in the list. It would save some code repetition, and avoid needing to change the `elif` condition, though the other part of my comment is that that seems strange to me in the first place (but, since it's outside the scope of this change, anyway, avoiding changing it at all seems better).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kwmonroe commented on the issue:

          https://github.com/apache/bigtop/pull/151

          @johnsca, `namenode_hosts` has only 1 item in non-HA mode. Note that once we hit the `else` to enter the "non HA" condition, we're not in any loop – `namenode_hosts[0]` is how the one and only namenode was being referenced in other parts of the template, so I kept it the same:

          https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml#L98

          Fwiw, I could use `@hadoop_namenode_host` if you think that would be cleaner.

          As for HA/Kerberos, I do not know why they are mutually exclusive; as you noted, I preserved the original logic.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kwmonroe commented on the issue: https://github.com/apache/bigtop/pull/151 @johnsca, `namenode_hosts` has only 1 item in non-HA mode. Note that once we hit the `else` to enter the "non HA" condition, we're not in any loop – `namenode_hosts [0] ` is how the one and only namenode was being referenced in other parts of the template, so I kept it the same: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml#L98 Fwiw, I could use `@hadoop_namenode_host` if you think that would be cleaner. As for HA/Kerberos, I do not know why they are mutually exclusive; as you noted, I preserved the original logic.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user johnsca commented on the issue:

          https://github.com/apache/bigtop/pull/151

          Would the `namenode_hosts` contain more than one item in non-HA? Perhaps it would be cleaner to just move the loop out of the conditional and rely on it only running once in the non-HA case.

          Also, why are HA and Kerberos mutually exclusive? The change preserves the existing logic, but the logic seems odd to me.

          Show
          githubbot ASF GitHub Bot added a comment - Github user johnsca commented on the issue: https://github.com/apache/bigtop/pull/151 Would the `namenode_hosts` contain more than one item in non-HA? Perhaps it would be cleaner to just move the loop out of the conditional and rely on it only running once in the non-HA case. Also, why are HA and Kerberos mutually exclusive? The change preserves the existing logic, but the logic seems odd to me.
          Hide
          kwmonroe Kevin W Monroe added a comment -

          This patch adds a block to configure ports in the hdfs-site.xml template when not in HA mode.

          Show
          kwmonroe Kevin W Monroe added a comment - This patch adds a block to configure ports in the hdfs-site.xml template when not in HA mode.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/151

          BIGTOP-2553: add rpc/http/https ports to template for non-HA mode

          Honor the `[namenode|http|https]_port` settings in non-HA mode.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop bug/BIGTOP-2553/non-HA-ports

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/151.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #151


          commit 6f05bee3deecd8ce8603230d1a16da1e5456a049
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2016-10-17T17:38:08Z

          add rpc/http/https ports to template for non-HA mode


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/151 BIGTOP-2553 : add rpc/http/https ports to template for non-HA mode Honor the ` [namenode|http|https] _port` settings in non-HA mode. You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop bug/ BIGTOP-2553 /non-HA-ports Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #151 commit 6f05bee3deecd8ce8603230d1a16da1e5456a049 Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2016-10-17T17:38:08Z add rpc/http/https ports to template for non-HA mode

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development