Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1806

Integrate Apache Ignite (incubating) with BigTop

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: build, debian, rpm
    • Labels:
      None

      Description

      We need to switch to the ASF version of this component.

      Provides Apache Ignite (incubating) BigTop build.
      The code is basically former "gridgain" build project adapted to Ignite project.

      1. BIGTOP-1806--d.patch
        41 kB
        Ivan Veselovsky
      2. BIGTOP-1806--c.patch
        38 kB
        Ivan Veselovsky
      3. IGNITE-BIGTOP--b.patch
        38 kB
        Ivan Veselovsky

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment -

          should we delete the gridgain build as part of this ?

          Show
          jayunit100 jay vyas added a comment - should we delete the gridgain build as part of this ?
          Hide
          evans_ye Evans Ye added a comment -

          I'm thinking the same. We should replace Gridgain by Ignite in 1.0

          Show
          evans_ye Evans Ye added a comment - I'm thinking the same. We should replace Gridgain by Ignite in 1.0
          Hide
          cos Konstantin Boudnik added a comment -

          I have looked over the patch and it seems that it is essentially a copy of what has been done for gridgain-hadoop component, so there's not much to comment upon, I think. Let's make remove all gridgain-hadoop parts either as a part of this patch, or better yet, make a subtask to remove them all. Then we'll be able to switch the two with minimal interruption. But first things first:

          • the build fails like this
            CONF_DIR=/etc/ignite-hadoop/conf.dist
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/lib/ignite-hadoop
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/lib/ignite-hadoop/libs
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/share/doc/ignite-hadoop-1.0.0
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/lib/ignite-hadoop/bin
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/lib/ignite-hadoop/bin/include
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//etc/ignite-hadoop
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//etc/ignite-hadoop/conf.dist
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/share/man/man1
            + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64/var/run/ignite-hadoop/work/
            + unzip -x 'target/incubator-ignite-*.zip'
            unzip:  cannot find or open target/incubator-ignite-*.zip, target/incubator-ignite-*.zip.zip or target/incubator-ignite-*.zip.ZIP.
            
            No zipfiles found.
            

          target directory contains however ignite-hadoop-1.0.0.zip zip file. Please fix it in the next iteration of the patch. Also, could you please test the patch before sending it for review? Please make sure that what is coming in is in a working condition as we are very close to the release here. Thanks!

          Show
          cos Konstantin Boudnik added a comment - I have looked over the patch and it seems that it is essentially a copy of what has been done for gridgain-hadoop component, so there's not much to comment upon, I think. Let's make remove all gridgain-hadoop parts either as a part of this patch, or better yet, make a subtask to remove them all. Then we'll be able to switch the two with minimal interruption. But first things first: the build fails like this CONF_DIR=/etc/ignite-hadoop/conf.dist + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/lib/ignite-hadoop + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/lib/ignite-hadoop/libs + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/share/doc/ignite-hadoop-1.0.0 + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/lib/ignite-hadoop/bin + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/lib/ignite-hadoop/bin/include + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//etc/ignite-hadoop + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//etc/ignite-hadoop/conf.dist + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64//usr/share/man/man1 + install -d -m 0755 /biggy/workspaces/bigtop/build/ignite-hadoop/rpm/BUILDROOT/ignite-hadoop-1.0.0-1.x86_64/var/run/ignite-hadoop/work/ + unzip -x 'target/incubator-ignite-*.zip' unzip: cannot find or open target/incubator-ignite-*.zip, target/incubator-ignite-*.zip.zip or target/incubator-ignite-*.zip.ZIP. No zipfiles found. target directory contains however ignite-hadoop-1.0.0.zip zip file. Please fix it in the next iteration of the patch. Also, could you please test the patch before sending it for review? Please make sure that what is coming in is in a working condition as we are very close to the release here. Thanks!
          Hide
          iveselovskiy Ivan Veselovsky added a comment -

          As per my understanding, yes, Ignite should replace GG in BigTop.

          Show
          iveselovskiy Ivan Veselovsky added a comment - As per my understanding, yes, Ignite should replace GG in BigTop.
          Hide
          iveselovskiy Ivan Veselovsky added a comment - - edited

          The patch was tested with complete build and installation to both Ubuntu (deb) and CentOS7 (rpm).

          Your build is failed because just before submitting the patch I changed the Git tag the sources are checked out by: that revision (1.0.0) really has a little bit different name than the tested one. Will fix that soon.

          Show
          iveselovskiy Ivan Veselovsky added a comment - - edited The patch was tested with complete build and installation to both Ubuntu (deb) and CentOS7 (rpm). Your build is failed because just before submitting the patch I changed the Git tag the sources are checked out by: that revision (1.0.0) really has a little bit different name than the tested one. Will fix that soon.
          Hide
          cos Konstantin Boudnik added a comment -

          Yes, and it produced ignite-hadoop-1.0.0.zip as expected.

          Show
          cos Konstantin Boudnik added a comment - Yes, and it produced ignite-hadoop-1.0.0.zip as expected.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          Yes, that's another part:

          +IGNITE_HADOOP_GIT_REPO=https://git-wip-us.apache.org/repos/asf/incubator-ignite
          +IGNITE_HADOOP_GIT_REF=ignite-1.0.0
          +IGNITE_HADOOP_GIT_DIR=$(IGNITE_HADOOP_NAME)-$(IGNITE_HADOOP_BASE_VERSION)
          +#IGNITE_HADOOP_VCS_TAG=release-$(IGNITE_HADOOP_BASE_VERSION)
          +IGNITE_HADOOP_RELEASE_VERSION=1
          +IGNITE_HADOOP_TARBALL_DST=$(IGNITE_HADOOP_NAME)-$(IGNITE_HADOOP_BASE_VERSION).tar.gz
          +IGNITE_HADOOP_TARBALL_SRC=$(IGNITE_HADOOP_NAME)-$(IGNITE_HADOOP_BASE_VERSION)-$(IGNITE_HADOOP_GIT_REF).tar.gz
          +#IGNITE_HADOOP_DOWNLOAD_PATH=http://apache-mirror.rbc.ru/pub/apache//incubator/ignite/1.0.0-RC3/
          +#IGNITE_HADOOP_DOWNLOAD_PATH=/ignite/ignite-hadoop-$(IGNITE_HADOOP_BASE_VERSION)
          +#IGNITE_HADOOP_SITE=$(IGNITE_HADOOP_DOWNLOAD_PATH)
          +#IGNITE_HADOOP_ARCHIVE=$(IGNITE_HADOOP_DOWNLOAD_PATH)
          

          Let's just use official 1.0.0 release, not the git tag - shall we?

          Show
          cos Konstantin Boudnik added a comment - - edited Yes, that's another part: +IGNITE_HADOOP_GIT_REPO=https://git-wip-us.apache.org/repos/asf/incubator-ignite +IGNITE_HADOOP_GIT_REF=ignite-1.0.0 +IGNITE_HADOOP_GIT_DIR=$(IGNITE_HADOOP_NAME)-$(IGNITE_HADOOP_BASE_VERSION) +#IGNITE_HADOOP_VCS_TAG=release-$(IGNITE_HADOOP_BASE_VERSION) +IGNITE_HADOOP_RELEASE_VERSION=1 +IGNITE_HADOOP_TARBALL_DST=$(IGNITE_HADOOP_NAME)-$(IGNITE_HADOOP_BASE_VERSION).tar.gz +IGNITE_HADOOP_TARBALL_SRC=$(IGNITE_HADOOP_NAME)-$(IGNITE_HADOOP_BASE_VERSION)-$(IGNITE_HADOOP_GIT_REF).tar.gz +#IGNITE_HADOOP_DOWNLOAD_PATH=http://apache-mirror.rbc.ru/pub/apache//incubator/ignite/1.0.0-RC3/ +#IGNITE_HADOOP_DOWNLOAD_PATH=/ignite/ignite-hadoop-$(IGNITE_HADOOP_BASE_VERSION) +#IGNITE_HADOOP_SITE=$(IGNITE_HADOOP_DOWNLOAD_PATH) +#IGNITE_HADOOP_ARCHIVE=$(IGNITE_HADOOP_DOWNLOAD_PATH) Let's just use official 1.0.0 release, not the git tag - shall we?
          Hide
          iveselovskiy Ivan Veselovsky added a comment -

          the tag "ignite-1.0.0" is pointing exactly to the official release 1.0.0.
          I have discussed this with Vladimir Ozerov – his opinion was that we should use the tag rather than zipped sources shared on the Apache download site.
          Let's discuss with Vladimir if there are other suggestions.

          Show
          iveselovskiy Ivan Veselovsky added a comment - the tag "ignite-1.0.0" is pointing exactly to the official release 1.0.0. I have discussed this with Vladimir Ozerov – his opinion was that we should use the tag rather than zipped sources shared on the Apache download site. Let's discuss with Vladimir if there are other suggestions.
          Hide
          cos Konstantin Boudnik added a comment -

          The way Bigtop creates stacks is by using official source artifacts. It isn't set in stone though, but that what we were doing for a long time. Perhaps it'd make sense to follow the lead, unless there's a good reason not to do this for Ignite?

          Show
          cos Konstantin Boudnik added a comment - The way Bigtop creates stacks is by using official source artifacts. It isn't set in stone though, but that what we were doing for a long time. Perhaps it'd make sense to follow the lead, unless there's a good reason not to do this for Ignite?
          Hide
          jayunit100 jay vyas added a comment -

          i think it weill be inevitable that folks in many cases, will need to build from a specific tag to deal w/ ecosystem interop in certain cases .... so its good to try tarball, but i think makes sense to totally allow folks to tag based build as well

          Show
          jayunit100 jay vyas added a comment - i think it weill be inevitable that folks in many cases, will need to build from a specific tag to deal w/ ecosystem interop in certain cases .... so its good to try tarball, but i think makes sense to totally allow folks to tag based build as well
          Hide
          cos Konstantin Boudnik added a comment -

          Fine by me. As I said - not set in stone

          Show
          cos Konstantin Boudnik added a comment - Fine by me. As I said - not set in stone
          Hide
          iveselovskiy Ivan Veselovsky added a comment - - edited

          patch "c":

          • fixed problem with the distro file name;
          • changed back to the Ignite source tarball download;

          Please note that some problems fixed in Ignite (IGNITE-665, IGNITE-346) are fixed in Ignite trunk, and are not present in 1.0.0 release.
          So, if Ignite BigTop will be build from 1.0.0 release, these fixes will not be available.

          Show
          iveselovskiy Ivan Veselovsky added a comment - - edited patch "c": fixed problem with the distro file name; changed back to the Ignite source tarball download; Please note that some problems fixed in Ignite ( IGNITE-665 , IGNITE-346 ) are fixed in Ignite trunk, and are not present in 1.0.0 release. So, if Ignite BigTop will be build from 1.0.0 release, these fixes will not be available.
          Hide
          cos Konstantin Boudnik added a comment -

          Patch builds now. One small thing: bigtop.mk still has some commented lines in - let's remove them and it seems to be ready to get committed. Also, could you add a ticket to remove the gridgain-hadoop component?

          Also, please format the patch according to https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
          Thanks!

          Show
          cos Konstantin Boudnik added a comment - Patch builds now. One small thing: bigtop.mk still has some commented lines in - let's remove them and it seems to be ready to get committed. Also, could you add a ticket to remove the gridgain-hadoop component? Also, please format the patch according to https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute Thanks!
          Hide
          iveselovskiy Ivan Veselovsky added a comment -

          Ticket to remove GridGain from BigTop: BIGTOP-1809 .
          Commented lines removed from bigtop.mk.
          Patch "d" created using the guidelines.

          Show
          iveselovskiy Ivan Veselovsky added a comment - Ticket to remove GridGain from BigTop: BIGTOP-1809 . Commented lines removed from bigtop.mk. Patch "d" created using the guidelines.
          Hide
          cos Konstantin Boudnik added a comment -

          Latest patch looks good. +1 - I will commit it in a bit.

          A hint for the future: you don't need to manually create patch-file suffixes: just keep the same name and JIRA will properly sort it out and mark the latest for easy navigation.

          Show
          cos Konstantin Boudnik added a comment - Latest patch looks good. +1 - I will commit it in a bit. A hint for the future: you don't need to manually create patch-file suffixes: just keep the same name and JIRA will properly sort it out and mark the latest for easy navigation.
          Hide
          cos Konstantin Boudnik added a comment -

          Pushed to the master. Thanks Ivan!

          Show
          cos Konstantin Boudnik added a comment - Pushed to the master. Thanks Ivan!
          Hide
          iveselovskiy Ivan Veselovsky added a comment -

          WRT the patch suffixes: 2 problems are there:
          1) if one would download a patch, he is risking to be confused on what exactly the downloaded patch is.
          2) if one refers the patch in comments , it is simplier to say "patch 'b'" than no say "patch of <date>" .

          Show
          iveselovskiy Ivan Veselovsky added a comment - WRT the patch suffixes: 2 problems are there: 1) if one would download a patch, he is risking to be confused on what exactly the downloaded patch is. 2) if one refers the patch in comments , it is simplier to say "patch 'b'" than no say "patch of <date>" .
          Hide
          jayunit100 jay vyas added a comment -

          for big patches that are iterated i highly suggest to use Github integration the way storm/spark and so many other projects are nowadays... makes iterative review so easy.

          Just make sure your commit has JIRA name i.e. "BIGTOP-1806. ...." so that the ASF Github daemon sees and attaches to the right project.

          just push commits so that people dont have to do patch diffs...

          it updates jira for you with a .patch link...

          when done, you can always create a final rebased patch if you want to !

          As an example , you can see BIGTOP-1812 that me and RJ Nowling just iterated on rapidly this morning.

          Show
          jayunit100 jay vyas added a comment - for big patches that are iterated i highly suggest to use Github integration the way storm/spark and so many other projects are nowadays... makes iterative review so easy. Just make sure your commit has JIRA name i.e. " BIGTOP-1806 . ...." so that the ASF Github daemon sees and attaches to the right project. just push commits so that people dont have to do patch diffs... it updates jira for you with a .patch link... when done, you can always create a final rebased patch if you want to ! As an example , you can see BIGTOP-1812 that me and RJ Nowling just iterated on rapidly this morning.
          Hide
          iveselovskiy Ivan Veselovsky added a comment -

          That works only if the contributor is able to push the changes into the repo. I cannot do. As I understand, many contributors cannot do too.

          Show
          iveselovskiy Ivan Veselovsky added a comment - That works only if the contributor is able to push the changes into the repo. I cannot do. As I understand, many contributors cannot do too.
          Hide
          cos Konstantin Boudnik added a comment -

          the use of github isn't mandatory by any means. I am not using the damn thing

          Show
          cos Konstantin Boudnik added a comment - the use of github isn't mandatory by any means. I am not using the damn thing
          Hide
          iveselovskiy Ivan Veselovsky added a comment -

          Konstantin, some more questions on Ignite-Hadoop integration:
          1) both deb and rpm packages have doc package, that is effectively empty. As per my observations in GridGain version those packages were empty also. What is the supposed content of those packages? Should we have empty doc packags at all?
          2) What about Ignite packages smoke tests? GG version had ./bigtop-tests/smoke-tests/gridgain-hadoop/ , + some entries in ./bigtop-tests/test-artifacts/package/src/main/resources/package_data.xml .
          3) What about Ignite puppet? GG had ./bigtop-deploy/puppet/modules/gridgain-hadoop/ .
          4) Should we add ignite-hadoop into Jenknins jobs creator ./bigtop-ci/jenkins/jobsCreator.groovy ?

          Show
          iveselovskiy Ivan Veselovsky added a comment - Konstantin, some more questions on Ignite-Hadoop integration: 1) both deb and rpm packages have doc package, that is effectively empty. As per my observations in GridGain version those packages were empty also. What is the supposed content of those packages? Should we have empty doc packags at all? 2) What about Ignite packages smoke tests? GG version had ./bigtop-tests/smoke-tests/gridgain-hadoop/ , + some entries in ./bigtop-tests/test-artifacts/package/src/main/resources/package_data.xml . 3) What about Ignite puppet? GG had ./bigtop-deploy/puppet/modules/gridgain-hadoop/ . 4) Should we add ignite-hadoop into Jenknins jobs creator ./bigtop-ci/jenkins/jobsCreator.groovy ?
          Hide
          cos Konstantin Boudnik added a comment -

          1. Fair question about doc package. Unless we want to populate it with real docs - which would make sense IMO - we can get rid of it (see BIGTOP-1818).
          2. Let's fix it. Please have a separate JIRA for that and link it here.
          3. Addressed by BIGTOP-1814. Please review
          4. Indeed. Again - as a separate JIRA.

          It'd be great to have all these, except for #1, done soon, as I am working on 1.0 RC now. Thanks!

          Show
          cos Konstantin Boudnik added a comment - 1. Fair question about doc package. Unless we want to populate it with real docs - which would make sense IMO - we can get rid of it (see BIGTOP-1818 ). 2. Let's fix it. Please have a separate JIRA for that and link it here. 3. Addressed by BIGTOP-1814 . Please review 4. Indeed. Again - as a separate JIRA. It'd be great to have all these, except for #1, done soon, as I am working on 1.0 RC now. Thanks!
          Hide
          iveselovskiy Ivan Veselovsky added a comment - - edited

          BIGTOP-1821 addresses smoke tests issue.

          Show
          iveselovskiy Ivan Veselovsky added a comment - - edited BIGTOP-1821 addresses smoke tests issue.
          Hide
          jayunit100 jay vyas added a comment - - edited

          NOBODY pushes to github, ever.

          but if you fork bigtop on github, and push to your own repo, you can open a pull request which will automatically attach patch link here.

          then you can push commits, and the patch is updated, and each individual commit (i.e. addressed the bigtop-1814
          issue) can be reviewed in isolation as the patch evolves.

          others use things like reviewboard and so on for this kind of thing... but thanks to asfbigtop git integration,
          we now get iterative review tooling integrated w jira *for free* .

          And *yes Cos is right* its not required by any means, and in the end, all patches get pushed to ASF Git not github.

          Sorry if i confused the issue, was trying to suggest a way to make your life easier .... hope it makes sense now.

          Show
          jayunit100 jay vyas added a comment - - edited NOBODY pushes to github, ever. but if you fork bigtop on github, and push to your own repo, you can open a pull request which will automatically attach patch link here. then you can push commits, and the patch is updated, and each individual commit (i.e. addressed the bigtop-1814 issue) can be reviewed in isolation as the patch evolves. others use things like reviewboard and so on for this kind of thing... but thanks to asfbigtop git integration, we now get iterative review tooling integrated w jira * for free * . And * yes Cos is right * its not required by any means, and in the end, all patches get pushed to ASF Git not github. Sorry if i confused the issue, was trying to suggest a way to make your life easier .... hope it makes sense now.

            People

            • Assignee:
              iveselovskiy Ivan Veselovsky
              Reporter:
              iveselovskiy Ivan Veselovsky
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development