Hadoop Common
  1. Hadoop Common
  2. HADOOP-4117

Improve configurability of Hadoop EC2 instances

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: contrib/cloud
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Changed scripts to pass initialization script for EC2 instances at boot time (as EC2 user data) rather than embedding initialization information in the EC2 image. This change makes it easy to customize the hadoop-site.xml file for your cluster before launch, by editing the hadoop-ec2-init-remote.sh script, or by setting the environment variable USER_DATA_FILE in hadoop-ec2-env.sh to run a script of your choice.
      Show
      Changed scripts to pass initialization script for EC2 instances at boot time (as EC2 user data) rather than embedding initialization information in the EC2 image. This change makes it easy to customize the hadoop-site.xml file for your cluster before launch, by editing the hadoop-ec2-init-remote.sh script, or by setting the environment variable USER_DATA_FILE in hadoop-ec2-env.sh to run a script of your choice.

      Description

      Currently hadoop-site.xml for EC2 instances is stored as a part of the image and only a few properties can be controlled from the user scripts (compression, number of map/reduce tasks). Furthermore, it is not possible to rsync the configuration around the EC2 cluster with the current image, so the only way to customize the hadoop-site.xml file is to rebuild the image, which is time-consuming.

      It would be much better to pass the initialization script for nodes at boot time, so that it is easy to edit the configuration before starting a cluster.

      1. hadoop-4117.patch
        15 kB
        Tom White
      2. hadoop-4117-v2.patch
        14 kB
        Tom White

        Activity

        Tom White created issue -
        Tom White made changes -
        Field Original Value New Value
        Attachment hadoop-4117.patch [ 12389697 ]
        Tom White made changes -
        Attachment hadoop-4117-v2.patch [ 12390301 ]
        Tom White made changes -
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Release Note Changed scripts to pass initialization script for EC2 instances at boot time (as EC2 user data) rather than embedding initialization information in the EC2 image. This change makes it easy to customize the hadoop-site.xml file for your cluster before launch, by editing the hadoop-ec2-init-remote.sh script, or by setting the environment variable USER_DATA_FILE in hadoop-ec2-env.sh to run a script of your choice.
        Status Open [ 1 ] Resolved [ 5 ]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development