Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1372

Bigtop needs feature that takes in multiple arguments to build specific components at command line

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:
      None

      Description

      Bigtop needs feature that takes in multiple arguments to build specific components at command line. For instance, "$ gradle hadoop hbase" will run tests for the artifacts: hadoop and hbase.

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment -

          Are you suggesting a higher level interface to the build for end users?

          Maybe that's a possibility but it might be more work than just adding a few options , it could become pretty complex as the functionality of the build gets more sophisticated, to maintain a high level interface to it.

          Show
          jayunit100 jay vyas added a comment - Are you suggesting a higher level interface to the build for end users? Maybe that's a possibility but it might be more work than just adding a few options , it could become pretty complex as the functionality of the build gets more sophisticated, to maintain a high level interface to it.
          Hide
          jeid Julien Eid added a comment -

          Can you give an example of what arguments you would want to give? Do you mean instead of doing "make hbase-deb; make hive-deb" you would do make hbase-deb hive-deb?

          Show
          jeid Julien Eid added a comment - Can you give an example of what arguments you would want to give? Do you mean instead of doing "make hbase-deb; make hive-deb" you would do make hbase-deb hive-deb?
          Hide
          dawson.choong Dawson Choong added a comment -

          Jay: Yes, on the current gradle interface, if I am not mistaken, only allows the option of all the test-artifacts being run ($ gradle installTestArtifacts). If one artifact is failing, we want to be able to test that single artifact without the rest of the tests running.

          Julien: Yes, that's correct. Something along the lines of "$ gradle hbase hadoop" which will launch tests for only those two artifacts.

          Show
          dawson.choong Dawson Choong added a comment - Jay: Yes, on the current gradle interface, if I am not mistaken, only allows the option of all the test-artifacts being run ($ gradle installTestArtifacts). If one artifact is failing, we want to be able to test that single artifact without the rest of the tests running. Julien: Yes, that's correct. Something along the lines of "$ gradle hbase hadoop" which will launch tests for only those two artifacts.
          Hide
          jayunit100 jay vyas added a comment -

          Ah, this is related to the tests ! There are two types of tests... we are currently moving the smoke tests out of the test-artifacts so that they are easy to customize and so that we can run them without building any jars.

          So there are:

          1) Maven based tests (bigtop-tests/test-artifacts)
          2) Gradle Functional tests (bigtop-smoke-tests/ being created in BIGTOP-1222)

          I think there is a good possibility that BIGTOP-1222 is exactly what you are looking for? That JIRA is the first iteration in cleaning up the bigtop tests to make them super easy and fast to run and use. I'd love your feedback on it and can help you set it up to test if you want to help in evaluating it.

          Show
          jayunit100 jay vyas added a comment - Ah, this is related to the tests ! There are two types of tests... we are currently moving the smoke tests out of the test-artifacts so that they are easy to customize and so that we can run them without building any jars. So there are: 1) Maven based tests (bigtop-tests/test-artifacts) 2) Gradle Functional tests (bigtop-smoke-tests/ being created in BIGTOP-1222 ) I think there is a good possibility that BIGTOP-1222 is exactly what you are looking for? That JIRA is the first iteration in cleaning up the bigtop tests to make them super easy and fast to run and use. I'd love your feedback on it and can help you set it up to test if you want to help in evaluating it.
          Hide
          apurtell Andrew Purtell added a comment -

          Ah, this is related to the tests

          Not just about tests, though, right? The gradle build can't build only one component at a time?

          Show
          apurtell Andrew Purtell added a comment - Ah, this is related to the tests Not just about tests, though, right? The gradle build can't build only one component at a time?
          Hide
          cos Konstantin Boudnik added a comment -

          Sorry Andre - are you asking if gradle can do parallel builds? If that was the question then the answer is resounding yes: if build is well-decoupled one can control the concurrency of execution with something like --parallel

          Show
          cos Konstantin Boudnik added a comment - Sorry Andre - are you asking if gradle can do parallel builds? If that was the question then the answer is resounding yes : if build is well-decoupled one can control the concurrency of execution with something like --parallel
          Hide
          apurtell Andrew Purtell added a comment -

          Never mind, I discovered 'gradle tasks --all'

          Show
          apurtell Andrew Purtell added a comment - Never mind, I discovered 'gradle tasks --all'
          Hide
          Virginia Wang Virginia Wang added a comment -

          Yes, exactly.

          Currently the only way to build the test artifact is run: gradle installAllLocalArtifacts or gradle installTestArtifacts which will install all of them. In some cases, we won't need some of the artifacts for example spark or hbase. An option should be added to only install a specific artifact, such as installArtifact hadoop.

          This allows the user to pick and choose instead of being forced to install all of them and having a risk of not supporting or using some of the artifacts.

          It is not so much relating to the tests as relating to a more flexible functionality in doing the installation with gradle.

          Show
          Virginia Wang Virginia Wang added a comment - Yes, exactly. Currently the only way to build the test artifact is run: gradle installAllLocalArtifacts or gradle installTestArtifacts which will install all of them. In some cases, we won't need some of the artifacts for example spark or hbase. An option should be added to only install a specific artifact, such as installArtifact hadoop. This allows the user to pick and choose instead of being forced to install all of them and having a risk of not supporting or using some of the artifacts. It is not so much relating to the tests as relating to a more flexible functionality in doing the installation with gradle.
          Hide
          cos Konstantin Boudnik added a comment -

          On top of what Virginia Wang just said.

          To clear up something from the legacy standpoint: the 'installAllLocalArtifacts' is an awful hack that I've done to get us going. If you look into how the packages.gradle is done you'll see a much more elegant build system that allows you do to 'all' or one or a few named packages build.

          I think it is only fair to have a somewhat similar functionality for test artifacts and test execution. I am also pretty sure - after implementing the packages build - that components discovery and tasks generation can be done dynamically, further reducing the need for maintenance.

          Show
          cos Konstantin Boudnik added a comment - On top of what Virginia Wang just said. To clear up something from the legacy standpoint: the 'installAllLocalArtifacts' is an awful hack that I've done to get us going. If you look into how the packages.gradle is done you'll see a much more elegant build system that allows you do to 'all' or one or a few named packages build. I think it is only fair to have a somewhat similar functionality for test artifacts and test execution. I am also pretty sure - after implementing the packages build - that components discovery and tasks generation can be done dynamically, further reducing the need for maintenance.
          Hide
          dawson.choong Dawson Choong added a comment -

          Thank you, Jay. Bigtop-1222 will help tremendously. I will take you up on your offer in helping me set it up for testing. I'll shoot you an email.

          Show
          dawson.choong Dawson Choong added a comment - Thank you, Jay. Bigtop-1222 will help tremendously. I will take you up on your offer in helping me set it up for testing. I'll shoot you an email.
          Hide
          rguo Guo Ruijing added a comment -

          Currently, build command is hard-coded in do-component-build. If we can pass parameters to do-component-build, we can use bigtop to do UT like:

          make hadoop-rpm $MVN_OPTION="test"

          Show
          rguo Guo Ruijing added a comment - Currently, build command is hard-coded in do-component-build. If we can pass parameters to do-component-build, we can use bigtop to do UT like: make hadoop-rpm $MVN_OPTION="test"
          Hide
          cos Konstantin Boudnik added a comment -

          make hadoop-rpm $MVN_OPTION="test"

          I actually don't like the idea of making build overly flexible. If one can pass arguments into do-component-build script it has a potential of affecting the content of produced build. Most obvious use case would be

          • having all dependencies prefetched and locally installed in .m2
          • passing in -o parameter to maven to prevent updates
          • resulting build might potentially has wrong deps built in

          Besides, it isn't a task of packaging part of the build to test anything. The assumption is that our source artifacts are already tested during their own release cycle. Bigtop community doesn't necessary have an expertise to test releases of all the components we bundle into the stack.

          Show
          cos Konstantin Boudnik added a comment - make hadoop-rpm $MVN_OPTION="test" I actually don't like the idea of making build overly flexible. If one can pass arguments into do-component-build script it has a potential of affecting the content of produced build. Most obvious use case would be having all dependencies prefetched and locally installed in .m2 passing in -o parameter to maven to prevent updates resulting build might potentially has wrong deps built in Besides, it isn't a task of packaging part of the build to test anything. The assumption is that our source artifacts are already tested during their own release cycle. Bigtop community doesn't necessary have an expertise to test releases of all the components we bundle into the stack.
          Hide
          cos Konstantin Boudnik added a comment -

          And I little bit confused: is there anything in the scope of the JIRA that BIGTOP-1222 (and its sequels) aren't covering? As stated in the description

          Bigtop needs feature that takes in multiple arguments to build specific components at command line.

          It has been done pretty much since day one, so you can something like gradle hadoop-rpm hbase-rpm and get yourself a neat two-components stack.

          Show
          cos Konstantin Boudnik added a comment - And I little bit confused: is there anything in the scope of the JIRA that BIGTOP-1222 (and its sequels) aren't covering? As stated in the description Bigtop needs feature that takes in multiple arguments to build specific components at command line. It has been done pretty much since day one, so you can something like gradle hadoop-rpm hbase-rpm and get yourself a neat two-components stack.
          Hide
          jayunit100 jay vyas added a comment -

          To clarify :

          • For tests: I think BIGTOP-1222 is specific for tests. this feature has indeed a root in the bigtop-tests/smoke-tests directory that can now be extended .
          • For building: Top level gradle tasks support no building selective components.

          So, as per those two bullets .... should we close this ticket?

          Show
          jayunit100 jay vyas added a comment - To clarify : For tests: I think BIGTOP-1222 is specific for tests. this feature has indeed a root in the bigtop-tests/smoke-tests directory that can now be extended . For building: Top level gradle tasks support no building selective components. So, as per those two bullets .... should we close this ticket?
          Hide
          cos Konstantin Boudnik added a comment -

          sounds good to me

          Show
          cos Konstantin Boudnik added a comment - sounds good to me
          Hide
          cos Konstantin Boudnik added a comment -

          a dup of BIGTOP-1222

          Show
          cos Konstantin Boudnik added a comment - a dup of BIGTOP-1222

            People

            • Assignee:
              Unassigned
              Reporter:
              dawson.choong Dawson Choong
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development