Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.5.0
    • Fix Version/s: None
    • Component/s: deployment
    • Labels:
      None

      Description

      We should have a tarball driver for each component, it will be:

      • convenient for people to deploy tarball to run map reduce job in a single VM
      • Whirr is only for EC2, but we want this tarball driver be generic for Linux user.
      • driver should deploy Hadoop tarball by default when deploy other tarball like Hive tarball, or Pig tarball
      • driver should be able to deploy multiple projects. For example, if user choose Hive+Pig, then driver will deploy Hadoop tarball + both Hive,Pig tarball
      • The version of each project tarball will be same as the version of project Bigtop 0.5 shipped

      I already have a prototype, and I am thinking about where to put it and a suitable way to invoke it as a user.

      1. BIGTOP-730.txt
        81 kB
        Johnny Zhang

        Activity

        Hide
        Roman Shaposhnik added a comment -

        Johnny, first of all don't hesitate to attach your prototype even if it doesn't integrate with Bigtop yet. It would be helpful for others to see what has been developed so far.

        I can immediately see the benefit in a tarball managemend backend for pmanager in iTest (as a sublcass of org.apache.bigtop.itest.pmanager.PackageInstance the same way things like DEBPackage/RPMPackage are its subclasses).

        Show
        Roman Shaposhnik added a comment - Johnny, first of all don't hesitate to attach your prototype even if it doesn't integrate with Bigtop yet. It would be helpful for others to see what has been developed so far. I can immediately see the benefit in a tarball managemend backend for pmanager in iTest (as a sublcass of org.apache.bigtop.itest.pmanager.PackageInstance the same way things like DEBPackage/RPMPackage are its subclasses).
        Hide
        Johnny Zhang added a comment -

        @Roman, thanks for the comments, I plan to add below classes

        in bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager

        class ManagedTar extends PackageInstance {
        ......
        public List<String> getFiles()
        ......
        public Map<String, Service> getServices()
        ......
        public List<String> getDocs()
        ......
        }
        

        and

        class TarCmdLinePackageManager extends PackageManager {
        ......
        public int install(PackageInstance pkg)
        ......
        public int remove(PackageInstance pkg)
        ......
        public boolean isInstalled(PackageInstance pkg)
        ......
        }
        

        also need to modify PackageInstance.groovy and PackageManager.groovy so that it can handle 4 types: yum, apt, zypper, tar

        in bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke

        TarTestCommon.groovy
        TestTarDeploy.groovy
        TestTarServices.groovy
        

        in bigtop-tests/test-artifacts/package/src/main/resources, add a new dir "tar" to include the tarball file manifest xml file

        Show
        Johnny Zhang added a comment - @Roman, thanks for the comments, I plan to add below classes in bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager class ManagedTar extends PackageInstance { ...... public List<String> getFiles() ...... public Map<String, Service> getServices() ...... public List<String> getDocs() ...... } and class TarCmdLinePackageManager extends PackageManager { ...... public int install(PackageInstance pkg) ...... public int remove(PackageInstance pkg) ...... public boolean isInstalled(PackageInstance pkg) ...... } also need to modify PackageInstance.groovy and PackageManager.groovy so that it can handle 4 types: yum, apt, zypper, tar in bigtop-tests/test-artifacts/package/src/main/groovy/org/apache/bigtop/itest/packagesmoke TarTestCommon.groovy TestTarDeploy.groovy TestTarServices.groovy in bigtop-tests/test-artifacts/package/src/main/resources, add a new dir "tar" to include the tarball file manifest xml file
        Hide
        Roman Shaposhnik added a comment -

        Looks nice in principle. Please post a patch (even if it is not ready for integration yet) so that we can all review it and provide more focused guidance.

        Show
        Roman Shaposhnik added a comment - Looks nice in principle. Please post a patch (even if it is not ready for integration yet) so that we can all review it and provide more focused guidance.
        Hide
        Johnny Zhang added a comment -

        @Roman, this is a design patch, it is a approve of concept (there are quite of new code as well as fix need to be put in). I have been able to to use run below command

        mvn -f pom.xml clean verify -Dorg.apache.bigtop.itest.log4j.level=TRACE -Dlog4j.debug=true -Dorg.apache.maven-failsafe-plugin.testInclude="**/TestTarBasics.*" -X
        

        to download the tarball, untar it, format NN and start NN/DN.

        Show
        Johnny Zhang added a comment - @Roman, this is a design patch, it is a approve of concept (there are quite of new code as well as fix need to be put in). I have been able to to use run below command mvn -f pom.xml clean verify -Dorg.apache.bigtop.itest.log4j.level=TRACE -Dlog4j.debug=true -Dorg.apache.maven-failsafe-plugin.testInclude="**/TestTarBasics.*" -X to download the tarball, untar it, format NN and start NN/DN.
        Hide
        Bruno Mahé added a comment -
        • Note that there quite a few references to Cloudera (or internal services of Cloudera) which should be removed. Also references to CDH should be replaced by references to Apache Bigtop.
        • In bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageInstance.groovy, I would rather put the RPM in its own case and error out for the default case.
        • Status command of the TarService is still using the service command
        • bigtop-tests/test-artifacts/package/src/main/resources/tar/config/hbase/hbase-site.xml doesn't seem right

        Other than that, this patch makes sense.

        Show
        Bruno Mahé added a comment - Note that there quite a few references to Cloudera (or internal services of Cloudera) which should be removed. Also references to CDH should be replaced by references to Apache Bigtop. In bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageInstance.groovy, I would rather put the RPM in its own case and error out for the default case. Status command of the TarService is still using the service command bigtop-tests/test-artifacts/package/src/main/resources/tar/config/hbase/hbase-site.xml doesn't seem right Other than that, this patch makes sense.
        Hide
        Johnny Zhang added a comment -

        Thanks for comments, Bruno Macherel, I will provide a new patch soon.

        Show
        Johnny Zhang added a comment - Thanks for comments, Bruno Macherel , I will provide a new patch soon.

          People

          • Assignee:
            Johnny Zhang
            Reporter:
            Johnny Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development