Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.8.0
    • Component/s: general
    • Labels:
      None

      Description

      Bigtop platform depends on groovy extensively and we see a trend to introduce new functionality into the cluster deployment, etc. that will greatly benefit from the presence of a dynamic, functional scripting language on top of JVM.

      Hence, it makes sense to provide Groovy runtime environment as a part of Bigtop installable stack (e.g. think of bigtop-utils type of package). It will be lightweight and only needs to include a couple of jar files from a standard Groovy distribution. There's no need to build Groovy from scratch but just download the binary archive, extract jars we need, and wrap them into a Linux package.

      1. BIGTOP-1097.patch
        8 kB
        Konstantin Boudnik
      2. BIGTOP-1097.patch
        14 kB
        Konstantin Boudnik
      3. BIGTOP-1097.patch
        15 kB
        Konstantin Boudnik
      4. BIGTOP-1097.patch
        15 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          The approach I will take is to essentially have a meta-package that will depend on whatever Groovy's supported and provided package out there...

          Show
          Konstantin Boudnik added a comment - The approach I will take is to essentially have a meta-package that will depend on whatever Groovy's supported and provided package out there...
          Hide
          Konstantin Boudnik added a comment -

          The problem though, is that most of packages for CENTOS and alike are real old: in the hood of Groovy 1.8, where the latest is 2.2 or even later.

          Show
          Konstantin Boudnik added a comment - The problem though, is that most of packages for CENTOS and alike are real old: in the hood of Groovy 1.8, where the latest is 2.2 or even later.
          Hide
          Konstantin Boudnik added a comment -

          With the above consideration, I think the better idea would be to actually build groovy as the part of our own stack and let it be installed along with the rest of the software. We don't need the full groovy environment - just runtime, essentially, e.g. groovy-<version>.jar; re-distribution will guarantee that we use the environment that we can rely upon, etc.

          Show
          Konstantin Boudnik added a comment - With the above consideration, I think the better idea would be to actually build groovy as the part of our own stack and let it be installed along with the rest of the software. We don't need the full groovy environment - just runtime, essentially, e.g. groovy-<version>.jar; re-distribution will guarantee that we use the environment that we can rely upon, etc.
          Hide
          Andrew Bayer added a comment -

          Do you really need 2.x features for the script in BIGTOP-952? That doesn't seem likely to me.

          Show
          Andrew Bayer added a comment - Do you really need 2.x features for the script in BIGTOP-952 ? That doesn't seem likely to me.
          Hide
          Konstantin Boudnik added a comment -

          It isn't about features, but bug fixes and performance. Besides, why sticking with odd-years old stuff if newer is available at no cost?

          Show
          Konstantin Boudnik added a comment - It isn't about features, but bug fixes and performance. Besides, why sticking with odd-years old stuff if newer is available at no cost?
          Hide
          Andrew Bayer added a comment -

          Well, there is a cost if you need to build/package it yourself, isn't there?

          Show
          Andrew Bayer added a comment - Well, there is a cost if you need to build/package it yourself, isn't there?
          Hide
          Konstantin Boudnik added a comment -

          Well, not exactly 'to build': just get the binary archive, extract and package a couple of files from it. But as the result we'll have a full control of the environment across all the OS spectra.

          Show
          Konstantin Boudnik added a comment - Well, not exactly 'to build': just get the binary archive, extract and package a couple of files from it. But as the result we'll have a full control of the environment across all the OS spectra.
          Hide
          Andrew Bayer added a comment -

          Ah, ok, I was afraid you meant building it from scratch, which seemed extremely over the top to me. =)

          Show
          Andrew Bayer added a comment - Ah, ok, I was afraid you meant building it from scratch, which seemed extremely over the top to me. =)
          Hide
          Konstantin Boudnik added a comment -

          Oh no - I am not that extreme I will update the description to make sure it reflect the approach correctly.

          Show
          Konstantin Boudnik added a comment - Oh no - I am not that extreme I will update the description to make sure it reflect the approach correctly.
          Hide
          jay vyas added a comment -

          Given that right now ALL of the groovy stuff in bigtop is for itest/smokes, which run via maven and automagically grab groovy for us:
          $> find ./ -name *groovy | grep -v "test-framework" | grep -v "test-artifacts" | wc -l
          0

          Adding groovy as something that bigtop repackages seems like adding complexity that doesnt really have any clear benefit. If we do so lets be very clear about the direction of things: Is bigtop moving towards using groovy as a core element of the core package build and deploying?

          As of now, groovy is only used for test-artifacts and test-execution and itest.

          Show
          jay vyas added a comment - Given that right now ALL of the groovy stuff in bigtop is for itest/smokes, which run via maven and automagically grab groovy for us: $> find ./ -name *groovy | grep -v "test-framework" | grep -v "test-artifacts" | wc -l 0 Adding groovy as something that bigtop repackages seems like adding complexity that doesnt really have any clear benefit. If we do so lets be very clear about the direction of things: Is bigtop moving towards using groovy as a core element of the core package build and deploying? As of now, groovy is only used for test-artifacts and test-execution and itest.
          Hide
          Konstantin Boudnik added a comment -

          Jay, this is a long going conversation and the reason we need Groovy is things like BIGTOP-952. Working with Java stack via shell-outs is very suboptimal to say the least. Instead we can do things directly via HDFS/MR/YARN/Hbase APIs if needed. I'd suggest to search through the dev@ archive on the discussion.

          Show
          Konstantin Boudnik added a comment - Jay, this is a long going conversation and the reason we need Groovy is things like BIGTOP-952 . Working with Java stack via shell-outs is very suboptimal to say the least. Instead we can do things directly via HDFS/MR/YARN/Hbase APIs if needed. I'd suggest to search through the dev@ archive on the discussion.
          Hide
          jay vyas added a comment -

          okay... so regarding this question:

          " Is bigtop moving towards using groovy as a core element of the core package build and deploying? "

          I guess the answer is : "Yes !"

          Show
          jay vyas added a comment - okay... so regarding this question: " Is bigtop moving towards using groovy as a core element of the core package build and deploying? " I guess the answer is : "Yes !"
          Hide
          Konstantin Boudnik added a comment -

          Right. That was the outcome of earlier discussion on the dev@ list.

          Show
          Konstantin Boudnik added a comment - Right. That was the outcome of earlier discussion on the dev@ list.
          Hide
          Sean Mackrory added a comment -

          I see that some of this was already committed. I'd argue for calling the package "bigtop-groovy" and not just "groovy" (I haven't checked if the latter is already in common use in other distros). It's not really a 1st-class citizen, it's more of a common dependency like bigtop-jsvc or bigtop-tomcat.

          Show
          Sean Mackrory added a comment - I see that some of this was already committed. I'd argue for calling the package "bigtop-groovy" and not just "groovy" (I haven't checked if the latter is already in common use in other distros). It's not really a 1st-class citizen, it's more of a common dependency like bigtop-jsvc or bigtop-tomcat.
          Hide
          Konstantin Boudnik added a comment -

          My bad, Sean - I accidentally pushed work in progress from my dev branch along with the other fixed. I have revoked that unfortunate commit. Appreciate the input on the naming - I was thinking along the same lines as well, as it needs to avoid clashes with native groovy package.

          Show
          Konstantin Boudnik added a comment - My bad, Sean - I accidentally pushed work in progress from my dev branch along with the other fixed. I have revoked that unfortunate commit. Appreciate the input on the naming - I was thinking along the same lines as well, as it needs to avoid clashes with native groovy package.
          Hide
          Roman Shaposhnik added a comment -

          Here's my belated 4 rubles:

          • +1 on sticking with Groovy 2.x
          • +1 on bigtop-groovy

          Finally, I think having a groovy launcher script would be extremely useful since I really would like to start doing #! env groovy

          Show
          Roman Shaposhnik added a comment - Here's my belated 4 rubles: +1 on sticking with Groovy 2.x +1 on bigtop-groovy Finally, I think having a groovy launcher script would be extremely useful since I really would like to start doing #! env groovy
          Hide
          Konstantin Boudnik added a comment - - edited

          First version of the package - RPM only right now - that pulls together fully functional version of groovy environment. One can use
          #!/usr/lib/bigtop-groovy/bin/groovy shebang directly in their scripts.

          DEB is coming. Feedbacks are welcome!

          Show
          Konstantin Boudnik added a comment - - edited First version of the package - RPM only right now - that pulls together fully functional version of groovy environment. One can use #!/usr/lib/bigtop-groovy/bin/groovy shebang directly in their scripts. DEB is coming. Feedbacks are welcome!
          Hide
          Konstantin Boudnik added a comment -

          Now with debian packaging.

          However, I hit a pretty nasty problem in our build system: turns out we can not build debian packages if the source stuff comes in the form of .zip file. How cool is that?

          I think there's a vulgar way to workaround it and repack an archive upfront. But I think it's better be fixed properly. I will open a ticket for it.

          In the meanwhile, if someone wants to test the patch, you can workaround the issue by repacking groovy-binary.zip to groovy-binary.tar.gz and adjusting bigtop.mk accordingly.

          Show
          Konstantin Boudnik added a comment - Now with debian packaging. However, I hit a pretty nasty problem in our build system: turns out we can not build debian packages if the source stuff comes in the form of .zip file. How cool is that? I think there's a vulgar way to workaround it and repack an archive upfront. But I think it's better be fixed properly. I will open a ticket for it. In the meanwhile, if someone wants to test the patch, you can workaround the issue by repacking groovy-binary.zip to groovy-binary.tar.gz and adjusting bigtop.mk accordingly.
          Hide
          Konstantin Boudnik added a comment -

          Ready for review.

          Show
          Konstantin Boudnik added a comment - Ready for review.
          Hide
          Konstantin Boudnik added a comment -

          Ah, one of the conveniences would be to add /usr/bin/bigtop-groovy link, so the shebang can be shorter, but I am sorta -0 on that.

          Show
          Konstantin Boudnik added a comment - Ah, one of the conveniences would be to add /usr/bin/bigtop-groovy link, so the shebang can be shorter, but I am sorta -0 on that.
          Hide
          Konstantin Boudnik added a comment -

          And now with package tests

          Show
          Konstantin Boudnik added a comment - And now with package tests
          Hide
          Konstantin Boudnik added a comment -

          Latest version of the patch that takes into consideration the changes from BIGTOP-1199.

          Show
          Konstantin Boudnik added a comment - Latest version of the patch that takes into consideration the changes from BIGTOP-1199 .
          Hide
          Roman Shaposhnik added a comment -

          LGTM. Please commit.

          Show
          Roman Shaposhnik added a comment - LGTM. Please commit.
          Hide
          Konstantin Boudnik added a comment -

          Pushed to master.

          Show
          Konstantin Boudnik added a comment - Pushed to master.

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development