Whirr
  1. Whirr
  2. WHIRR-5

Run namenode and jobtracker on separate EC2 instances

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: contrib/python
    • Labels:
      None

      Description

      Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.

        Activity

        Hide
        Matei Zaharia added a comment -

        Just curious, what's the reasoning for this? I would imagine that the NN communicates with the JT a lot and thus it helps to colocate them. Another useful option might be to allow the user to ask for a bigger instance type for the "master" node (if the script doesn't currently allow this).

        Show
        Matei Zaharia added a comment - Just curious, what's the reasoning for this? I would imagine that the NN communicates with the JT a lot and thus it helps to colocate them. Another useful option might be to allow the user to ask for a bigger instance type for the "master" node (if the script doesn't currently allow this).
        Hide
        Tom White added a comment -

        Colocation is the default, but splitting the namenode/jobtracker is better for large clusters. Good point about using a larger instance for the master - I've opened HADOOP-6393 for this.

        Show
        Tom White added a comment - Colocation is the default, but splitting the namenode/jobtracker is better for large clusters. Good point about using a larger instance for the master - I've opened HADOOP-6393 for this.
        Hide
        Tom White added a comment -

        Here's a patch that removes the strong requirement to have "master" and "slave" types. Instead, instances have one or more roles, which are stored in EC2 as security groups. The instance uses knowledge of its role to start the relevant daemons at boot time - for example the "nn" role will start up the namenode.

        Here's how you launch one master node (running a namenode, secondary namenode, and a jobtracker) and 10 datanodes/tasktrackers:

        hadoop-ec2 launch-cluster my-hadoop-cluster 1 nn,snn,jt 10 dn,tt
        

        The following command will launch a namenode and a separate jobtracker (and 10 datanodes/tasktrackers).

        hadoop-ec2 launch-cluster my-hadoop-cluster 1 nn,snn 1 jt 10 dn,tt
        

        This patch is fairly large, but there is little new code, since it is mainly refactoring. Previously, the logic for each command was spread between cli.py and commands.py. Now, cli.py is responsible only for command line parsing, and it delegates to service.py, which contains the class HadoopService.py for the Hadoop-specific logic.

        Show
        Tom White added a comment - Here's a patch that removes the strong requirement to have "master" and "slave" types. Instead, instances have one or more roles, which are stored in EC2 as security groups. The instance uses knowledge of its role to start the relevant daemons at boot time - for example the "nn" role will start up the namenode. Here's how you launch one master node (running a namenode, secondary namenode, and a jobtracker) and 10 datanodes/tasktrackers: hadoop-ec2 launch-cluster my-hadoop-cluster 1 nn,snn,jt 10 dn,tt The following command will launch a namenode and a separate jobtracker (and 10 datanodes/tasktrackers). hadoop-ec2 launch-cluster my-hadoop-cluster 1 nn,snn 1 jt 10 dn,tt This patch is fairly large, but there is little new code, since it is mainly refactoring. Previously, the logic for each command was spread between cli.py and commands.py. Now, cli.py is responsible only for command line parsing, and it delegates to service.py, which contains the class HadoopService.py for the Hadoop-specific logic.
        Hide
        Aaron Kimball added a comment -

        +1 looks good to me.

        Show
        Aaron Kimball added a comment - +1 looks good to me.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427398/HADOOP-6392.patch
        against trunk revision 888565.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427398/HADOOP-6392.patch against trunk revision 888565. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/console This message is automatically generated.
        Hide
        Tom White added a comment -

        I'll commit this tomorrow unless there are any objections.

        Show
        Tom White added a comment - I'll commit this tomorrow unless there are any objections.
        Hide
        Tom White added a comment -

        I've just committed this.

        Show
        Tom White added a comment - I've just committed this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #102 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/102/)
        . Run namenode and jobtracker on separate EC2 instances.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #102 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/102/ ) . Run namenode and jobtracker on separate EC2 instances.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #183 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/183/)
        . Run namenode and jobtracker on separate EC2 instances.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #183 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/183/ ) . Run namenode and jobtracker on separate EC2 instances.

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development