Bigtop
  1. Bigtop
  2. BIGTOP-1310

Vagrant recipes provision 2.0 hadoop w/ puppet recipes that are compatible with 2.2

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I'm noticing in the Vagrant recipe, that if i do

      vagrant up bigtop1
      

      I get an error :

      "Invalid shuffle port number -1"

      I then have to change the do

        <value>mapreduce.shuffle</value>   
      

      And it works !

      I looked further and saw that BIGTOP-1118 actually changed the value , for hadoop 2.2.

      So this means hadoop 2.0 is NOT compatible with bigtop puppet recipes, which are built for hadoop 2.2+ anymore.

      1) We need to gaurd puppet recipes: They should only be applied to the appropriate hadoop version. Is there a way to do this?

      2) We need to upgrade vagrant incantations to provison current bigtop if theres a way to do that, or at least hard code them to 2.2 , rather than 2.0.

        Activity

        Hide
        jay vyas added a comment -

        this is resolved after 0.8

        Show
        jay vyas added a comment - this is resolved after 0.8
        Hide
        Konstantin Boudnik added a comment -

        I am +1 on closing it!

        Show
        Konstantin Boudnik added a comment - I am +1 on closing it!
        Hide
        Evans Ye added a comment -

        This jira has been fixed by BIGTOP-1449 (The default bigtop repo has been updated to 0.8.0, hence the hadoop version is 2.4.1 now).
        Shall we just close it as fixed, jay vyas?

        Show
        Evans Ye added a comment - This jira has been fixed by BIGTOP-1449 (The default bigtop repo has been updated to 0.8.0, hence the hadoop version is 2.4.1 now). Shall we just close it as fixed, jay vyas ?
        Hide
        Evans Ye added a comment - - edited

        Hi jay vyas. Great documentation!
        Just a little bit thought on it:
        1) It might be better to move step 6(update yarn-site.xml) before step 5(provisioning) because if someone want to add a component into a provisioned machine, re-run step 5 will be required and after doing so the yarn-site.xml will be replaced back by template. Explicitly tell user to update yarn-site.xml template before provisioning can avoid that happening.
        2) It seems that step 5's shell command has a little bit typo:

        puppet -d --modulepath=/opt/bigtop/bigtop-deploy/puppet/modu[root@localhost puppet]# puppet -d --modulepath=/opt/bigtop/bigtop-deploy/puppet/modules --confdir=/opt/bigtop/bigtop-deploy/puppet/ /opt/bigtop/bigtop-deploy/puppet/manifests/site.pp
        

        Pointing to SNAPSHOT is good for me since this can fix the broken yarn provisioning ASAP

        Show
        Evans Ye added a comment - - edited Hi jay vyas . Great documentation! Just a little bit thought on it: 1) It might be better to move step 6(update yarn-site.xml) before step 5(provisioning) because if someone want to add a component into a provisioned machine, re-run step 5 will be required and after doing so the yarn-site.xml will be replaced back by template. Explicitly tell user to update yarn-site.xml template before provisioning can avoid that happening. 2) It seems that step 5's shell command has a little bit typo: puppet -d --modulepath=/opt/bigtop/bigtop-deploy/puppet/modu[root@localhost puppet]# puppet -d --modulepath=/opt/bigtop/bigtop-deploy/puppet/modules --confdir=/opt/bigtop/bigtop-deploy/puppet/ /opt/bigtop/bigtop-deploy/puppet/manifests/site.pp Pointing to SNAPSHOT is good for me since this can fix the broken yarn provisioning ASAP
        Hide
        jay vyas added a comment - - edited

        Okay, update on this:

        original issue of using the new shuffle parameter in yarn-site.xml for older hadoop versions: the final step in this wiki page (changing the mapreduce.aux_services.shuffle parameter, and then restarting yarn services) is a sufficient workaround to hold us over to bigtop 0.8.0 release. https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet.

        next steps: we will CLOSE this jira by UPDATING the recipes to point to SNAPSHOT when BIGTOP-1323 is done and then jenkins build is fixed. agreed ?

        Show
        jay vyas added a comment - - edited Okay, update on this: original issue of using the new shuffle parameter in yarn-site.xml for older hadoop versions : the final step in this wiki page (changing the mapreduce.aux_services.shuffle parameter, and then restarting yarn services) is a sufficient workaround to hold us over to bigtop 0.8.0 release. https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet . next steps: we will CLOSE this jira by UPDATING the recipes to point to SNAPSHOT when BIGTOP-1323 is done and then jenkins build is fixed. agreed ?
        Hide
        Roman Shaposhnik added a comment -

        Evans Ye understood. keep and eye on the build dockerization JIRA BIGTOP-1323 I'm almost done with the phase 0 and should be posting some major updates end of next week. Once that is done you'll be able to get a very reliable build env onto your laptop and then you can use that to repro any kind of issues that happen on our Jenkins.

        Show
        Roman Shaposhnik added a comment - Evans Ye understood. keep and eye on the build dockerization JIRA BIGTOP-1323 I'm almost done with the phase 0 and should be posting some major updates end of next week. Once that is done you'll be able to get a very reliable build env onto your laptop and then you can use that to repro any kind of issues that happen on our Jenkins.
        Hide
        Evans Ye added a comment -

        Would love to help on unscrewing builds. But I don't have access right on it and might need some time to get familiar with bigtop's jenkins settings. I would say feel free to call me up if you need a help.

        Show
        Evans Ye added a comment - Would love to help on unscrewing builds. But I don't have access right on it and might need some time to get familiar with bigtop's jenkins settings. I would say feel free to call me up if you need a help.
        Hide
        Konstantin Boudnik added a comment -

        Roman Shaposhnik, for hbase build issues please see this comment I've left on BIGTOP-1110 - it seems to be related to a need of JDK7

        Show
        Konstantin Boudnik added a comment - Roman Shaposhnik , for hbase build issues please see this comment I've left on BIGTOP-1110 - it seems to be related to a need of JDK7
        Hide
        Roman Shaposhnik added a comment -

        Konstantin Boudnik it is highly unlikely that ALL of them are screwed up in the same way and the build error is the same across all of them.

        Regardless, you're right – we need to get this under control, otherwise I don't think 0.8.0 is happening any time soon. I've unscrewed quite a bit of what was wrong over the weekend (Hadoop now builds for example, etc.). Anybody volunteering to unscrew HBase build?

        Show
        Roman Shaposhnik added a comment - Konstantin Boudnik it is highly unlikely that ALL of them are screwed up in the same way and the build error is the same across all of them. Regardless, you're right – we need to get this under control, otherwise I don't think 0.8.0 is happening any time soon. I've unscrewed quite a bit of what was wrong over the weekend (Hadoop now builds for example, etc.). Anybody volunteering to unscrew HBase build?
        Hide
        Konstantin Boudnik added a comment -

        From what I've seen in the HBase build errors: it doesn't look like HBase build problem per se. After all, I was able to run the build of Hadoop + Hbase on a number of time in different CI setups. I think Evans is right - some of the slaves are screwed up and need some love.

        Show
        Konstantin Boudnik added a comment - From what I've seen in the HBase build errors: it doesn't look like HBase build problem per se. After all, I was able to run the build of Hadoop + Hbase on a number of time in different CI setups. I think Evans is right - some of the slaves are screwed up and need some love.
        Hide
        Evans Ye added a comment -

        Okay, thanks for providing the new build #270. Just as Roman Shaposhnik said, hbase build doesn't up to date. Right now the jenkins repo can not provision a well running hadoop cluster due to the protobuf version between hadoop(2.5.0) and hbase(2.4.0a) are not compatible. In order to sort this out I think fixing those jenkins builds will be something we need to deal with first.
        Shall we prioritize current existing builds in to groups? Maybe put trunk builds into first priority group and we aim on fixing them as our first stage to clean up our CI. After that we then decide a bunch of builds into second priority group and so on. Just like jay vyas said that we probably will not going to fix those low priority or long failing builds. In such a case we just wiped them out. Would this be a possible solution?

        Show
        Evans Ye added a comment - Okay, thanks for providing the new build #270. Just as Roman Shaposhnik said, hbase build doesn't up to date. Right now the jenkins repo can not provision a well running hadoop cluster due to the protobuf version between hadoop(2.5.0) and hbase(2.4.0a) are not compatible. In order to sort this out I think fixing those jenkins builds will be something we need to deal with first. Shall we prioritize current existing builds in to groups? Maybe put trunk builds into first priority group and we aim on fixing them as our first stage to clean up our CI. After that we then decide a bunch of builds into second priority group and so on. Just like jay vyas said that we probably will not going to fix those low priority or long failing builds. In such a case we just wiped them out. Would this be a possible solution?
        Hide
        jay vyas added a comment -

        maybe the fastest route to a green build is decimating alot of the old builds that are red ?

        Sooner we clean up easier for others to pitch in and maintain — see comment in BIGTOP-1286 (can we gut all these old projects on jenkins so its easier to assess, contribute to, and maintain the bigtop builds by newcomers ?).

        Show
        jay vyas added a comment - maybe the fastest route to a green build is decimating alot of the old builds that are red ? Sooner we clean up easier for others to pitch in and maintain — see comment in BIGTOP-1286 (can we gut all these old projects on jenkins so its easier to assess, contribute to, and maintain the bigtop builds by newcomers ?).
        Hide
        Roman Shaposhnik added a comment -

        jay vyas Evans Ye in general our nightly builds are 100% deployable from these repos: http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/ in the usual apt/yum/zypper way. The repos/list files are located under each platform's lastSuccessfulBuild. For example, here's where the repo file for CentOS6 is: http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos6/lastSuccessfulBuild/artifact/repo/

        That's the theory. The practice, however, is that our build system was wedged for quite some time now and some of the components haven't been updated in those nightly repos. Hadoop, for example, is ok, but HBase is not. It would be really nice if we can get to a full green build ASAP. Your help would be more than welcome.

        Show
        Roman Shaposhnik added a comment - jay vyas Evans Ye in general our nightly builds are 100% deployable from these repos: http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/ in the usual apt/yum/zypper way. The repos/list files are located under each platform's lastSuccessfulBuild. For example, here's where the repo file for CentOS6 is: http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos6/lastSuccessfulBuild/artifact/repo/ That's the theory. The practice, however, is that our build system was wedged for quite some time now and some of the components haven't been updated in those nightly repos. Hadoop, for example, is ok, but HBase is not. It would be really nice if we can get to a full green build ASAP. Your help would be more than welcome.
        Hide
        jay vyas added a comment -

        Maybe Konstantin Boudnik or Sean Mackrory can chime in here.. is it possible to grab the 0.8.0-snapshots from the build server ? or do those not go out until 0.8.0 release.

        I dont know how this stuff works (yet)

        Show
        jay vyas added a comment - Maybe Konstantin Boudnik or Sean Mackrory can chime in here.. is it possible to grab the 0.8.0-snapshots from the build server ? or do those not go out until 0.8.0 release. I dont know how this stuff works (yet)
        Hide
        Evans Ye added a comment -

        Hi Jay.
        I think the way you suggested is good since we'd like to fix the broken YARN provisioning asap and leave the improvement task as a further development.
        I'd love to help on the patch, but as I know currently we don't have the latest bigtop packages for 0.8.0-snapshot, which means that bigtop 0.7.0(hadoop-2.0.6-alpha) is the only choice now. Maybe we can have new packages available by triggering the bigtop jenkins repo build? Sorry I don't have much knowledge about bigtop jenkins.

        Show
        Evans Ye added a comment - Hi Jay. I think the way you suggested is good since we'd like to fix the broken YARN provisioning asap and leave the improvement task as a further development. I'd love to help on the patch, but as I know currently we don't have the latest bigtop packages for 0.8.0-snapshot, which means that bigtop 0.7.0(hadoop-2.0.6-alpha) is the only choice now. Maybe we can have new packages available by triggering the bigtop jenkins repo build? Sorry I don't have much knowledge about bigtop jenkins.
        Hide
        jay vyas added a comment -

        Hi evans.

        My vote is to hard code it to 2.2 do you have time to do that and submit the patch ? Then I'll review it and then you can open another JIRA to upgrade with your facter ideas ?

        I'd love to upgrade to facter - but i think its better to do the simple fix in this JIRA

        Show
        jay vyas added a comment - Hi evans. My vote is to hard code it to 2.2 do you have time to do that and submit the patch ? Then I'll review it and then you can open another JIRA to upgrade with your facter ideas ? I'd love to upgrade to facter - but i think its better to do the simple fix in this JIRA
        Hide
        Evans Ye added a comment -

        I also agree that vagrant provisioner should stick together with current bigtop version. However I don't know much about the repository currently we use:

        http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
        

        I just grep it from the repo file in bigtop installation guide.

        To fix this, as long as we can always build and upload the newest rpm up to the S3 repository, it should be easy. We can just hard code the snapshot repository.
        Another way is to fulfill the compatibility between 2.0 and 2.2 by puppet recipes.
        We can write a puppet facter to auto detect hadoop version and use simple if-else in yarn-site.xml template to determine which value we should place.
        Last one , just like jay vyas said, may be we can just hard code them to 2.2, and I can run through a test on puppet recipes against the new hadoop version.

        Show
        Evans Ye added a comment - I also agree that vagrant provisioner should stick together with current bigtop version. However I don't know much about the repository currently we use: http: //bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64 I just grep it from the repo file in bigtop installation guide . To fix this, as long as we can always build and upload the newest rpm up to the S3 repository, it should be easy. We can just hard code the snapshot repository. Another way is to fulfill the compatibility between 2.0 and 2.2 by puppet recipes. We can write a puppet facter to auto detect hadoop version and use simple if-else in yarn-site.xml template to determine which value we should place. Last one , just like jay vyas said, may be we can just hard code them to 2.2, and I can run through a test on puppet recipes against the new hadoop version.
        Hide
        jay vyas added a comment -

        FYI, there is a mailing list thread about this for deciding how to deal with this, thanks mark for pointing this out ! http://mail-archives.apache.org/mod_mbox/bigtop-user/201405.mbox/%3CCAAu13zFgnhoq=pREB8oD7MsVy6a=1+5yYsHjNuDmSLpL=S8F7A@mail.gmail.com%3E

        Show
        jay vyas added a comment - FYI, there is a mailing list thread about this for deciding how to deal with this, thanks mark for pointing this out ! http://mail-archives.apache.org/mod_mbox/bigtop-user/201405.mbox/%3CCAAu13zFgnhoq=pREB8oD7MsVy6a=1+5yYsHjNuDmSLpL=S8F7A@mail.gmail.com%3E

          People

          • Assignee:
            jay vyas
            Reporter:
            jay vyas
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development