Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1052 HDFS scalability with multiple namenodes
  3. HDFS-1703

HDFS federation: Improve start/stop scripts and add script to decommission datanodes

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: Federation Branch
    • Fix Version/s: Federation Branch, 0.23.0
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      The masters file is no longer used to indicate which hosts to start the 2NN on. The 2NN is now started on hosts when dfs.namenode.secondary.http-address is configured with a non-wildcard IP.

      Description

      This Jira covers two issues:

      1. Startup scripts should start namenodes, secondary namenodes and datanodes on hosts retunred by getConfig (new feature). This patch is spread out to both common(HADOOP-7179) and hdfs (this Jira).
      2. Decommission script to decommission datanodes
      1. HDFS-1703.patch
        11 kB
        Tanping Wang

        Issue Links

          Activity

          Tanping Wang created issue -
          Hide
          Tanping Wang added a comment -

          Patch,
          HDFS-1703.patch-common
          goes to common.

          Show
          Tanping Wang added a comment - Patch, HDFS-1703 .patch-common goes to common.
          Tanping Wang made changes -
          Field Original Value New Value
          Attachment HDFS-1703.patch-common [ 12472477 ]
          Hide
          Tanping Wang added a comment -

          patch, HDFS-1703.patch-hdfs goes to hdfs.

          Show
          Tanping Wang added a comment - patch, HDFS-1703 .patch-hdfs goes to hdfs.
          Tanping Wang made changes -
          Attachment HDFS-1703.patch-hdfs [ 12472478 ]
          Hide
          Tanping Wang added a comment -

          Contributed by Erik Steffl.

          Show
          Tanping Wang added a comment - Contributed by Erik Steffl.
          Tanping Wang made changes -
          Link This issue is blocked by HDFS-1730 [ HDFS-1730 ]
          Tanping Wang made changes -
          Link This issue is blocked by HDFS-1730 [ HDFS-1730 ]
          Tanping Wang made changes -
          Link This issue is blocked by HADOOP-7179 [ HADOOP-7179 ]
          Tanping Wang made changes -
          Attachment HDFS-1703.patch-common [ 12472477 ]
          Hide
          Tanping Wang added a comment -

          I just created HADOOP-7179 and attached the change that needs to go to COMMON there.

          Show
          Tanping Wang added a comment - I just created HADOOP-7179 and attached the change that needs to go to COMMON there.
          Tanping Wang made changes -
          Summary HDFS federation: Start namenodes and secondary namenodes on hosts returned by hdfs getconf HDFS federation: Improve start/stop scripts and add script to decommission datanodes
          Description Startup scripts should start namenodes, secondary namenodes and datanodes on hosts retunred by getConfig (new feature). This patch is spread out to both common and hdfs. This Jira covers two issues:

          # Startup scripts should start namenodes, secondary namenodes and datanodes on hosts retunred by getConfig (new feature). This patch is spread out to both common(HADOOP-7179) and hdfs (this Jira).
          # Decommission script to decommission datanodes
          Hide
          Tanping Wang added a comment -

          HDFS-1730-decommission.patch contains the decommission scripts to decommission datanodes. This covers item #2 from Jira description.

          Show
          Tanping Wang added a comment - HDFS-1730 -decommission.patch contains the decommission scripts to decommission datanodes. This covers item #2 from Jira description.
          Tanping Wang made changes -
          Attachment HDFS-1730-decommission.patch [ 12473334 ]
          Tanping Wang made changes -
          Attachment HDFS-1703.patch-hdfs [ 12472478 ]
          Hide
          Tanping Wang added a comment -

          HDFS-1703-StartStop.patch converts item #1 from the Jira description.

          Show
          Tanping Wang added a comment - HDFS-1703 -StartStop.patch converts item #1 from the Jira description.
          Tanping Wang made changes -
          Attachment HDFS-1703-StartStop.patch [ 12473340 ]
          Hide
          Suresh Srinivas added a comment -

          Can you please combine the patches. Also a patch is prefixed with HDFS-1730, instead of HDFS-1703.

          Show
          Suresh Srinivas added a comment - Can you please combine the patches. Also a patch is prefixed with HDFS-1730 , instead of HDFS-1703 .
          Tanping Wang made changes -
          Attachment HDFS-1703-StartStop.patch [ 12473340 ]
          Tanping Wang made changes -
          Attachment HDFS-1730-decommission.patch [ 12473334 ]
          Hide
          Tanping Wang added a comment -

          The new startup script has the following changed behavior:

          1. Old behavior:
            • startup script had to be run on primary namenode. The script started namenode on the the node where it was run. Secondary namenode was started on the node in "master" file.
            • Datanodes were started on nodes that are in "slaves" file.
          2. New behavior:
            • Script determines namenodes and secondary namenodes from the hdfs-site.xml configuration and starts appropriate nodes on these machines. The file "master" is no longer required.(Changed behavior)
            • Datanodes were started on nodes that are in "slaves" file. (No change)

          Installation using their own scripts can start the nodes, based on the example in this script.

          Decommission tool:

          1. Two new scripts are added for performing decommissioning in federation setup.

          Combined all the scripts change into one patch, HDFS-1703.patch

          Show
          Tanping Wang added a comment - The new startup script has the following changed behavior: Old behavior: startup script had to be run on primary namenode. The script started namenode on the the node where it was run. Secondary namenode was started on the node in "master" file. Datanodes were started on nodes that are in "slaves" file. New behavior: Script determines namenodes and secondary namenodes from the hdfs-site.xml configuration and starts appropriate nodes on these machines. The file "master" is no longer required.(Changed behavior) Datanodes were started on nodes that are in "slaves" file. (No change) Installation using their own scripts can start the nodes, based on the example in this script. Decommission tool: Two new scripts are added for performing decommissioning in federation setup. Combined all the scripts change into one patch, HDFS-1703 .patch
          Tanping Wang made changes -
          Attachment HDFS-1703.patch [ 12473347 ]
          Hide
          Suresh Srinivas added a comment -

          Updated description:

          The new startup script has the following changed behavior:

          1. Old behavior:
            • startup script had to be run on primary namenode. The script started namenode on the the node where it was run. Secondary namenode was started on the node in "master" file.
            • Datanodes were started on nodes that are in "slaves" file.
          2. New behavior:
            • Script determines namenodes and secondary namenodes from the hdfs-site.xml configuration and starts appropriate nodes on these machines. The file "master" is no longer required.(Changed behavior)
            • Datanodes were started on nodes that are in "slaves" file. (No change)

          Installation using their own scripts can start the nodes, based on the example in this script.

          Decommission tool:

          1. Two new scripts are added for performing decommissioning in federation setup. This could be used for setup with single namenode also.
          2. distribute-exclude.sh
            • This script copies the given exclude file based to all the namenodes (discovered from the configuration hdfs-site.xml) to the exclude file as definied in the hdfs-site.xml configuration.
          3. refresh-namenodes.sh
            • This script runs refreshNodes command on all the namenodes (discovered from the configuration hdfs-site.xml).
          Show
          Suresh Srinivas added a comment - Updated description: The new startup script has the following changed behavior: Old behavior: startup script had to be run on primary namenode. The script started namenode on the the node where it was run. Secondary namenode was started on the node in "master" file. Datanodes were started on nodes that are in "slaves" file. New behavior: Script determines namenodes and secondary namenodes from the hdfs-site.xml configuration and starts appropriate nodes on these machines. The file "master" is no longer required.(Changed behavior) Datanodes were started on nodes that are in "slaves" file. (No change) Installation using their own scripts can start the nodes, based on the example in this script. Decommission tool: Two new scripts are added for performing decommissioning in federation setup. This could be used for setup with single namenode also. distribute-exclude.sh This script copies the given exclude file based to all the namenodes (discovered from the configuration hdfs-site.xml) to the exclude file as definied in the hdfs-site.xml configuration. refresh-namenodes.sh This script runs refreshNodes command on all the namenodes (discovered from the configuration hdfs-site.xml).
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Suresh Srinivas added a comment -

          Committed the patch. Thank you Tanping and Erik.

          Show
          Suresh Srinivas added a comment - Committed the patch. Thank you Tanping and Erik.
          Suresh Srinivas made changes -
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          Matt Foley added a comment -

          Merged to trunk on Fri Apr 29 18:16:32 2011 UTC, as part of merge of umbrella Jira HDFS-1052. (suresh)

          Show
          Matt Foley added a comment - Merged to trunk on Fri Apr 29 18:16:32 2011 UTC, as part of merge of umbrella Jira HDFS-1052 . (suresh)
          Matt Foley made changes -
          Fix Version/s 0.23.0 [ 12315571 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Eli Collins added a comment -

          Marking as incompatible and adding a release note.

          Show
          Eli Collins added a comment - Marking as incompatible and adding a release note.
          Eli Collins made changes -
          Hadoop Flags Reviewed [ 10343 ] Incompatible change,Reviewed [ 10342,10343 ]
          Release Note The masters file is no longer used to indicate which hosts to start the 2NN on. The 2NN is now started on hosts when dfs.namenode.secondary.http-address is configured with a non-wildcard IP.
          Eli Collins made changes -
          Link This issue is related to HDFS-2893 [ HDFS-2893 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          8d 1h 50m 1 Suresh Srinivas 11/Mar/11 00:04
          Resolved Resolved Closed Closed
          249d 47m 1 Arun C Murthy 15/Nov/11 00:52

            People

            • Assignee:
              Tanping Wang
              Reporter:
              Tanping Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development