Whirr
  1. Whirr
  2. WHIRR-9

Support additional security group option in hadoop-ec2 script

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: contrib/python
    • Labels:
      None

      Description

      When deploying a hadoop cluster on ec2 alongside other services it is very useful to be able to specify additional (pre-existing) security groups to facilitate access control. For example one could use this feature to add a cluster to a generic "hadoop" group, which authorizes hdfs access from instances outside the cluster. Without such an option the access control for the security groups created by the script need to manually updated after cluster launch.

      1. hadoop-ec2-py-0.3.0.patch
        5 kB
        Paul Egan
      2. hadoop-trunk-contrib-cloud.patch
        6 kB
        Paul Egan
      3. hadoop-trunk-contrib-cloud.patch
        6 kB
        Paul Egan
      4. hadoop-trunk-contrib-cloud.patch
        6 kB
        Paul Egan

        Activity

        Hide
        Paul Egan added a comment -

        Patch against cloudera-for-hadoop-on-ec2-py-0.3.0-beta.tar.gz adding a security-group option.

        Show
        Paul Egan added a comment - Patch against cloudera-for-hadoop-on-ec2-py-0.3.0-beta.tar.gz adding a security-group option.
        Hide
        Paul Egan added a comment -

        Patch against contrib/cloud adding a security-group option.

        Show
        Paul Egan added a comment - Patch against contrib/cloud adding a security-group option.
        Hide
        Tom White added a comment -

        Security groups are an EC2-only feature so it would be good to document that in the option help (just like --key-name, for example). Otherwise, this looks good.

        Show
        Tom White added a comment - Security groups are an EC2-only feature so it would be good to document that in the option help (just like --key-name, for example). Otherwise, this looks good.
        Hide
        Paul Egan added a comment -

        Updated help text for --security-group option.

        Show
        Paul Egan added a comment - Updated help text for --security-group option.
        Hide
        Paul Egan added a comment -

        Updated patch as requested

        Show
        Paul Egan added a comment - Updated patch as requested
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428594/hadoop-trunk-contrib-cloud.patch
        against trunk revision 892113.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/227/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428594/hadoop-trunk-contrib-cloud.patch against trunk revision 892113. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/227/console This message is automatically generated.
        Hide
        Tom White added a comment -

        Paul, thanks for the updated patch. Hudson can't apply it since it needs to be relative to Hadoop Common's root - i.e. paths starting with "src/contrib/cloud/src/py/hadoop/..." Could you regenerate it please? (BTW you can leave this issue assigned to yourself since you are working on it.)

        Show
        Tom White added a comment - Paul, thanks for the updated patch. Hudson can't apply it since it needs to be relative to Hadoop Common's root - i.e. paths starting with "src/contrib/cloud/src/py/hadoop/..." Could you regenerate it please? (BTW you can leave this issue assigned to yourself since you are working on it.)
        Hide
        Paul Egan added a comment -

        Updated patch to latest trunk and corrected paths

        Show
        Paul Egan added a comment - Updated patch to latest trunk and corrected paths
        Hide
        Paul Egan added a comment -

        No tests provided because an ec2 account with existing security groups would be required.

        Tested manually by running the hadoop-ec2 command with the new --security-group option and verified that the cluster nodes have been added to the specified group.

        Show
        Paul Egan added a comment - No tests provided because an ec2 account with existing security groups would be required. Tested manually by running the hadoop-ec2 command with the new --security-group option and verified that the cluster nodes have been added to the specified group.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428692/hadoop-trunk-contrib-cloud.patch
        against trunk revision 892113.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/230/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/230/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/230/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/230/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428692/hadoop-trunk-contrib-cloud.patch against trunk revision 892113. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/230/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/230/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/230/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/230/console This message is automatically generated.
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Paul!

        Show
        Tom White added a comment - I've just committed this. Thanks Paul!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #123 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/123/)
        . Support additional security group option in hadoop-ec2 script. Contributed by Paul Egan.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #123 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/123/ ) . Support additional security group option in hadoop-ec2 script. Contributed by Paul Egan.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/196/)
        . Support additional security group option in hadoop-ec2 script. Contributed by Paul Egan.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/196/ ) . Support additional security group option in hadoop-ec2 script. Contributed by Paul Egan.

          People

          • Assignee:
            Paul Egan
            Reporter:
            Paul Egan
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development