Whirr
  1. Whirr
  2. WHIRR-696

Whirr script for Hadoop MRv2 YARN installs that supports Hadoop-2.0.x and Hadoop 3.x (trunk) branches

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.2
    • Component/s: None
    • Labels:
      None

      Description

      The existing yarn recipes don't configure YARN properly for the elephant's trunk (Hadoop-2.0.x and 3.x), just Hadoop 0.23 version.

      This recipe makes it work out of the box, just:

      bin/whirr launch-cluster --config recipies/yarn-trunk-ec2.properties

      Yay!

      1. WHIRR-696-1.patch
        4 kB
        Eli Reisman
      2. WHIRR-696-2.patch
        3 kB
        Eli Reisman
      3. WHIRR-696-3.patch
        3 kB
        Eli Reisman
      4. WHIRR-696-4.patch
        3 kB
        Eli Reisman

        Activity

        Hide
        Eli Reisman added a comment -

        Thanks, that was a really helpful article! I'll put up a JIRA for it.

        Show
        Eli Reisman added a comment - Thanks, that was a really helpful article! I'll put up a JIRA for it.
        Hide
        Tom White added a comment -

        2.0.2-alpha will not drop the 'alpha' label, so there's no need to wait, unless we wait for 2.0.3-alpha which is being voted on now.

        BTW there is a discussion about the version labelling here, since it is pretty confusing!

        http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201301.mbox/%3CF4A9CA29-6536-4F3D-95B4-9BF18F14C8DE@hortonworks.com%3E

        Show
        Tom White added a comment - 2.0.2-alpha will not drop the 'alpha' label, so there's no need to wait, unless we wait for 2.0.3-alpha which is being voted on now. BTW there is a discussion about the version labelling here, since it is pretty confusing! http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201301.mbox/%3CF4A9CA29-6536-4F3D-95B4-9BF18F14C8DE@hortonworks.com%3E
        Hide
        Eli Reisman added a comment -

        I can put up a new jira and fix this, but if 2.0.3 comes out in alpha,
        2.0.2 will probably drop the "alpha" label. Perhaps we should wait to
        update this config until the name change on the mirrors? Or we can go with
        2.0.1 for now? Either way, Whirr seems to run well with 2.0.2-alpha so its
        a viable fix.

        Show
        Eli Reisman added a comment - I can put up a new jira and fix this, but if 2.0.3 comes out in alpha, 2.0.2 will probably drop the "alpha" label. Perhaps we should wait to update this config until the name change on the mirrors? Or we can go with 2.0.1 for now? Either way, Whirr seems to run well with 2.0.2-alpha so its a viable fix.
        Hide
        Tom White added a comment -

        We should track the latest 2 release, since YARN is still changing quickly as it stabilizes. It's OK if we don't continue to support older 2 releases. 2.0.2-alpha is the latest, but 2.0.3-alpha will be out very soon.

        Show
        Tom White added a comment - We should track the latest 2 release, since YARN is still changing quickly as it stabilizes. It's OK if we don't continue to support older 2 releases. 2.0.2-alpha is the latest, but 2.0.3-alpha will be out very soon.
        Hide
        Eli Reisman added a comment -

        Yeah, I have been running 2.0.2-alpha myself but I think if its OK with you we should migrate quickly to 2.x in this recipe. Which version do you think? 2.0.0? Anyway, yes this template works well with 2.0.x

        Show
        Eli Reisman added a comment - Yeah, I have been running 2.0.2-alpha myself but I think if its OK with you we should migrate quickly to 2.x in this recipe. Which version do you think? 2.0.0? Anyway, yes this template works well with 2.0.x
        Hide
        Tom White added a comment -

        +1 I committed this. Thanks Eli!

        We probably want the default to be a 2.x release, since the 0.23 releases are being driven by Yahoo!, mainly for their use. That can be done in another JIRA though, since this is already an improvement. Did you try it with 2.x in the end?

        Show
        Tom White added a comment - +1 I committed this. Thanks Eli! We probably want the default to be a 2.x release, since the 0.23 releases are being driven by Yahoo!, mainly for their use. That can be done in another JIRA though, since this is already an improvement. Did you try it with 2.x in the end?
        Hide
        Eli Reisman added a comment -

        Slight change. Realized that 23.1 was the original default hadoop to load for YARN. Then looked at the mirror we use. I will keep this set to 23.5 as the default, because this mirror does not serve 23.1 any more. This might be part of the Hadoop service install problem. Perhaps its time to look at changing it to a mirror that will list all versions of Hadoop?

        Anyway, this works and is good to go now. Thanks again!

        Show
        Eli Reisman added a comment - Slight change. Realized that 23.1 was the original default hadoop to load for YARN. Then looked at the mirror we use. I will keep this set to 23.5 as the default, because this mirror does not serve 23.1 any more. This might be part of the Hadoop service install problem. Perhaps its time to look at changing it to a mirror that will list all versions of Hadoop? Anyway, this works and is good to go now. Thanks again!
        Hide
        Eli Reisman added a comment -

        This is the cleaned up version, run just now fresh on EC2 and works fine. on us-east-1 and still works on 0.23.5 hadoop as well.

        Show
        Eli Reisman added a comment - This is the cleaned up version, run just now fresh on EC2 and works fine. on us-east-1 and still works on 0.23.5 hadoop as well.
        Hide
        Tom White added a comment -

        > I can do one more tweak on this to clean it up a little and to return some of the config values to previous settings (i.e. I assume you want it to default to hadoop 0.23 and AWS region us-east-1?) if you like.

        That would be great.

        Show
        Tom White added a comment - > I can do one more tweak on this to clean it up a little and to return some of the config values to previous settings (i.e. I assume you want it to default to hadoop 0.23 and AWS region us-east-1?) if you like. That would be great.
        Hide
        Eli Reisman added a comment -

        I can do one more tweak on this to clean it up a little and to return some of the config values to previous settings (i.e. I assume you want it to default to hadoop 0.23 and AWS region us-east-1?) if you like.

        I will put up another JIRA on the JDK issue, and maybe a patch to go with

        thanks!

        Show
        Eli Reisman added a comment - I can do one more tweak on this to clean it up a little and to return some of the config values to previous settings (i.e. I assume you want it to default to hadoop 0.23 and AWS region us-east-1?) if you like. I will put up another JIRA on the JDK issue, and maybe a patch to go with thanks!
        Hide
        Tom White added a comment -

        This looks fine to me. I can commit it if there are no further changes.

        > The Java install on newer Debian systems seems to die in the install script at a call to update-alternatives to set the default Java JDK. It should be an easy fix.

        Would you like to open another JIRA for that? Thanks!

        Show
        Tom White added a comment - This looks fine to me. I can commit it if there are no further changes. > The Java install on newer Debian systems seems to die in the install script at a call to update-alternatives to set the default Java JDK. It should be an easy fix. Would you like to open another JIRA for that? Thanks!
        Hide
        Eli Reisman added a comment -

        This consolidates the changes into the hadoop-yarn-ec2 recipe. I hardcoded a stable version of OS platform as there is a problem with the java installation in the hadoop service that should (and will) have its own JIRA. This patch works on EC2.

        The Java install on newer Debian systems seems to die in the install script at a call to update-alternatives to set the default Java JDK. It should be an easy fix.

        Show
        Eli Reisman added a comment - This consolidates the changes into the hadoop-yarn-ec2 recipe. I hardcoded a stable version of OS platform as there is a problem with the java installation in the hadoop service that should (and will) have its own JIRA. This patch works on EC2. The Java install on newer Debian systems seems to die in the install script at a call to update-alternatives to set the default Java JDK. It should be an easy fix.
        Hide
        Eli Reisman added a comment -

        Sounds good, I'll submit another patch. I was not certain if I should alter existing configs for 0.23.

        Show
        Eli Reisman added a comment - Sounds good, I'll submit another patch. I was not certain if I should alter existing configs for 0.23.
        Hide
        Tom White added a comment -

        Thanks for this Eli. I think it would be better to update hadoop-yarn-ec2.properties so we don't have duplicate recipes files.

        Show
        Tom White added a comment - Thanks for this Eli. I think it would be better to update hadoop-yarn-ec2.properties so we don't have duplicate recipes files.

          People

          • Assignee:
            Eli Reisman
            Reporter:
            Eli Reisman
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development