Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2893

The start/stop scripts don't start/stop the 2NN when using the default configuration

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 0.23.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-1703 changed the behavior of the start/stop scripts so that the masters file is no longer used to indicate which hosts to start the 2NN on. The 2NN is now started, when using start-dfs.sh, on hosts only when dfs.namenode.secondary.http-address is configured with a non-wildcard IP. This means you can not start a NN using an http-address specified using a wildcard IP. We should allow a 2NN to be started with the default config, ie start-dfs.sh should start a NN, 2NN and DN. The packaging already works this way (it doesn't use start-dfs.sh, it uses hadoop-daemon.sh directly w/o first checking getconf) so let's bring start-dfs.sh in line with this behavior.

      1. hdfs-2893.txt
        4 kB
        Eli Collins

        Issue Links

          Activity

          Eli Collins created issue -
          Eli Collins made changes -
          Field Original Value New Value
          Summary The 2NN won't start if dfs.namenode.secondary.http-address is default or specified with a wildcard IP and port start-dfs.sh won't start the 2NN if dfs.namenode.secondary.http-address is default or specified with a wildcard IP
          Assignee Eli Collins [ eli2 ]
          Description Looks like DFSUtil address matching doesn't find a match if the http-address is specified using a wildcard IP and a port. It should return 0.0.0.0:50090 in this case which would allow the 2NN to start.

          Also, unless http-address is explicitly configured in hdfs-site.xml the 2NN will not start, since DFSUtil#getSecondaryNameNodeAddresses does not use the default value as a fallback. That may be confusing to people who expect the default value to be used.

          {noformat}
          hadoop-0.23.1-SNAPSHOT $ cat /home/eli/hadoop/conf3/hdfs-site.xml
          ...
            <property>
              <name>dfs.namenode.secondary.http-address</name>
              <value>0.0.0.0:50090</value>
            </property>
          </configuration>

          hadoop-0.23.1-SNAPSHOT $ ./bin/hdfs --config ~/hadoop/conf3 getconf -secondarynamenodes
          0.0.0.0
          hadoop-0.23.1-SNAPSHOT $ ./sbin/start-dfs.sh
          Starting namenodes on [localhost]
          localhost: starting namenode, logging to /home/eli/hadoop/dirs3/logs/eli/hadoop-eli-namenode-eli-thinkpad.out
          localhost: starting datanode, logging to /home/eli/hadoop/dirs3/logs/eli/hadoop-eli-datanode-eli-thinkpad.out
          Secondary namenodes are not configured. Cannot start secondary namenodes.
          {noformat}

          This works if eg localhost:50090 is used.

          We should also update the hdfs user guide to remove the reference to the masters file since it's no longer used to configure which hosts the 2NN runs on.
          HDFS-1703 changed the behavior of the start/stop scripts so that the masters file is no longer used to indicate which hosts to start the 2NN on. The 2NN is now started, when using start-dfs.sh, on hosts only when dfs.namenode.secondary.http-address is configured with a non-wildcard IP. This means you can not start a NN using an http-address specified using a wildcard IP. We should allow a 2NN to be started with the default config, ie start-dfs.sh should start a NN, 2NN and DN. The packaging already works this way (it doesn't use start-dfs.sh, it uses hadoop-daemon.sh directly w/o first checking getconf) so let's bring start-dfs.sh in line with this behavior.

          Eli Collins made changes -
          Attachment hdfs-2893.txt [ 12513376 ]
          Eli Collins made changes -
          Link This issue relates to HDFS-1703 [ HDFS-1703 ]
          Eli Collins made changes -
          Summary start-dfs.sh won't start the 2NN if dfs.namenode.secondary.http-address is default or specified with a wildcard IP The start and stop scripts should start the 2NN when using the default configuration
          Priority Critical [ 2 ] Minor [ 4 ]
          Eli Collins made changes -
          Summary The start and stop scripts should start the 2NN when using the default configuration The start/stop scripts don't start/stop the 2NN when using the default configuration
          Eli Collins made changes -
          Link This issue relates to HDFS-2894 [ HDFS-2894 ]
          Eli Collins made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 0.23.1 [ 12318885 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HDFS-2995 [ HDFS-2995 ]
          Eli Collins made changes -
          Assignee Eli Collins [ eli2 ] Eli Collins [ eli ]

            People

            • Assignee:
              Eli Collins
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development