Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.8.0
    • Component/s: general
    • Labels:
      None

      Description

      Bigtop platform depends on groovy extensively and we see a trend to introduce new functionality into the cluster deployment, etc. that will greatly benefit from the presence of a dynamic, functional scripting language on top of JVM.

      Hence, it makes sense to provide Groovy runtime environment as a part of Bigtop installable stack (e.g. think of bigtop-utils type of package). It will be lightweight and only needs to include a couple of jar files from a standard Groovy distribution. There's no need to build Groovy from scratch but just download the binary archive, extract jars we need, and wrap them into a Linux package.

      1. BIGTOP-1097.patch
        8 kB
        Konstantin Boudnik
      2. BIGTOP-1097.patch
        14 kB
        Konstantin Boudnik
      3. BIGTOP-1097.patch
        15 kB
        Konstantin Boudnik
      4. BIGTOP-1097.patch
        15 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          117d 5h 39m 1 Konstantin Boudnik 21/Jan/14 01:44
          In Progress In Progress Patch Available Patch Available
          9d 4h 38m 1 Konstantin Boudnik 30/Jan/14 06:23
          Patch Available Patch Available Resolved Resolved
          12d 17h 14m 1 Konstantin Boudnik 11/Feb/14 23:37
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Konstantin Boudnik added a comment -

          Pushed to master.

          Show
          Konstantin Boudnik added a comment - Pushed to master.
          Hide
          Roman Shaposhnik added a comment -

          LGTM. Please commit.

          Show
          Roman Shaposhnik added a comment - LGTM. Please commit.
          Konstantin Boudnik made changes -
          Attachment BIGTOP-1097.patch [ 12628320 ]
          Hide
          Konstantin Boudnik added a comment -

          Latest version of the patch that takes into consideration the changes from BIGTOP-1199.

          Show
          Konstantin Boudnik added a comment - Latest version of the patch that takes into consideration the changes from BIGTOP-1199 .
          Konstantin Boudnik made changes -
          Attachment BIGTOP-1097.patch [ 12626267 ]
          Hide
          Konstantin Boudnik added a comment -

          And now with package tests

          Show
          Konstantin Boudnik added a comment - And now with package tests
          Hide
          Konstantin Boudnik added a comment -

          Ah, one of the conveniences would be to add /usr/bin/bigtop-groovy link, so the shebang can be shorter, but I am sorta -0 on that.

          Show
          Konstantin Boudnik added a comment - Ah, one of the conveniences would be to add /usr/bin/bigtop-groovy link, so the shebang can be shorter, but I am sorta -0 on that.
          Konstantin Boudnik made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Hide
          Konstantin Boudnik added a comment -

          Ready for review.

          Show
          Konstantin Boudnik added a comment - Ready for review.
          Konstantin Boudnik made changes -
          Link This issue is blocked by BIGTOP-1199 [ BIGTOP-1199 ]
          Konstantin Boudnik made changes -
          Attachment BIGTOP-1097.patch [ 12626057 ]
          Hide
          Konstantin Boudnik added a comment -

          Now with debian packaging.

          However, I hit a pretty nasty problem in our build system: turns out we can not build debian packages if the source stuff comes in the form of .zip file. How cool is that?

          I think there's a vulgar way to workaround it and repack an archive upfront. But I think it's better be fixed properly. I will open a ticket for it.

          In the meanwhile, if someone wants to test the patch, you can workaround the issue by repacking groovy-binary.zip to groovy-binary.tar.gz and adjusting bigtop.mk accordingly.

          Show
          Konstantin Boudnik added a comment - Now with debian packaging. However, I hit a pretty nasty problem in our build system: turns out we can not build debian packages if the source stuff comes in the form of .zip file. How cool is that? I think there's a vulgar way to workaround it and repack an archive upfront. But I think it's better be fixed properly. I will open a ticket for it. In the meanwhile, if someone wants to test the patch, you can workaround the issue by repacking groovy-binary.zip to groovy-binary.tar.gz and adjusting bigtop.mk accordingly.
          Konstantin Boudnik made changes -
          Comment [ I just checked Fedora 15 and the output of service status command is like
          {noformat}
          [root@localhost ~]# service crond status
          Redirecting to /bin/systemctl status crond.service
          crond.service - Command Scheduler
                    Loaded: loaded (/lib/systemd/system/crond.service)
                    Active: active (running) since Thu, 31 Oct 2013 08:05:50 -0400; 2 months and 29 days ago
                  Main PID: 853 (crond)
                    CGroup: name=systemd:/system/crond.service
                            └ 853 /usr/sbin/crond -n
          {noformat}
          So, 2nd line won't work on it too.

          I'd resort to use exit codes perhaps? ]
          Konstantin Boudnik made changes -
          Attachment BIGTOP-1097.patch [ 12626038 ]
          Hide
          Konstantin Boudnik added a comment - - edited

          First version of the package - RPM only right now - that pulls together fully functional version of groovy environment. One can use
          #!/usr/lib/bigtop-groovy/bin/groovy shebang directly in their scripts.

          DEB is coming. Feedbacks are welcome!

          Show
          Konstantin Boudnik added a comment - - edited First version of the package - RPM only right now - that pulls together fully functional version of groovy environment. One can use #!/usr/lib/bigtop-groovy/bin/groovy shebang directly in their scripts. DEB is coming. Feedbacks are welcome!
          Hide
          Roman Shaposhnik added a comment -

          Here's my belated 4 rubles:

          • +1 on sticking with Groovy 2.x
          • +1 on bigtop-groovy

          Finally, I think having a groovy launcher script would be extremely useful since I really would like to start doing #! env groovy

          Show
          Roman Shaposhnik added a comment - Here's my belated 4 rubles: +1 on sticking with Groovy 2.x +1 on bigtop-groovy Finally, I think having a groovy launcher script would be extremely useful since I really would like to start doing #! env groovy
          Hide
          Konstantin Boudnik added a comment -

          My bad, Sean - I accidentally pushed work in progress from my dev branch along with the other fixed. I have revoked that unfortunate commit. Appreciate the input on the naming - I was thinking along the same lines as well, as it needs to avoid clashes with native groovy package.

          Show
          Konstantin Boudnik added a comment - My bad, Sean - I accidentally pushed work in progress from my dev branch along with the other fixed. I have revoked that unfortunate commit. Appreciate the input on the naming - I was thinking along the same lines as well, as it needs to avoid clashes with native groovy package.
          Hide
          Sean Mackrory added a comment -

          I see that some of this was already committed. I'd argue for calling the package "bigtop-groovy" and not just "groovy" (I haven't checked if the latter is already in common use in other distros). It's not really a 1st-class citizen, it's more of a common dependency like bigtop-jsvc or bigtop-tomcat.

          Show
          Sean Mackrory added a comment - I see that some of this was already committed. I'd argue for calling the package "bigtop-groovy" and not just "groovy" (I haven't checked if the latter is already in common use in other distros). It's not really a 1st-class citizen, it's more of a common dependency like bigtop-jsvc or bigtop-tomcat.
          Konstantin Boudnik made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          Konstantin Boudnik added a comment -

          Right. That was the outcome of earlier discussion on the dev@ list.

          Show
          Konstantin Boudnik added a comment - Right. That was the outcome of earlier discussion on the dev@ list.
          Hide
          jay vyas added a comment -

          okay... so regarding this question:

          " Is bigtop moving towards using groovy as a core element of the core package build and deploying? "

          I guess the answer is : "Yes !"

          Show
          jay vyas added a comment - okay... so regarding this question: " Is bigtop moving towards using groovy as a core element of the core package build and deploying? " I guess the answer is : "Yes !"
          Hide
          Konstantin Boudnik added a comment -

          Jay, this is a long going conversation and the reason we need Groovy is things like BIGTOP-952. Working with Java stack via shell-outs is very suboptimal to say the least. Instead we can do things directly via HDFS/MR/YARN/Hbase APIs if needed. I'd suggest to search through the dev@ archive on the discussion.

          Show
          Konstantin Boudnik added a comment - Jay, this is a long going conversation and the reason we need Groovy is things like BIGTOP-952 . Working with Java stack via shell-outs is very suboptimal to say the least. Instead we can do things directly via HDFS/MR/YARN/Hbase APIs if needed. I'd suggest to search through the dev@ archive on the discussion.
          Hide
          jay vyas added a comment -

          Given that right now ALL of the groovy stuff in bigtop is for itest/smokes, which run via maven and automagically grab groovy for us:
          $> find ./ -name *groovy | grep -v "test-framework" | grep -v "test-artifacts" | wc -l
          0

          Adding groovy as something that bigtop repackages seems like adding complexity that doesnt really have any clear benefit. If we do so lets be very clear about the direction of things: Is bigtop moving towards using groovy as a core element of the core package build and deploying?

          As of now, groovy is only used for test-artifacts and test-execution and itest.

          Show
          jay vyas added a comment - Given that right now ALL of the groovy stuff in bigtop is for itest/smokes, which run via maven and automagically grab groovy for us: $> find ./ -name *groovy | grep -v "test-framework" | grep -v "test-artifacts" | wc -l 0 Adding groovy as something that bigtop repackages seems like adding complexity that doesnt really have any clear benefit. If we do so lets be very clear about the direction of things: Is bigtop moving towards using groovy as a core element of the core package build and deploying? As of now, groovy is only used for test-artifacts and test-execution and itest.
          Konstantin Boudnik made changes -
          Description Bigtop platform depends on groovy extensively and we see a trend to introduce new functionality into the cluster deployment, etc. that will greatly benefit from the presence of a dynamic, functional scripting language on top of JVM.

          Hence, it makes sense to provide Groovy runtime environment as a part of Bigtop installable stack (e.g. think of bigtop-utils type of package). It will be lightweight and only needs to include a couple of jar files from a standard Groovy distribution. There's no need to build Groovy from scratch but just download the binary archive, extract jars we need, and wrap them into a Linux package.
          Hide
          Konstantin Boudnik added a comment -

          Oh no - I am not that extreme I will update the description to make sure it reflect the approach correctly.

          Show
          Konstantin Boudnik added a comment - Oh no - I am not that extreme I will update the description to make sure it reflect the approach correctly.
          Hide
          Andrew Bayer added a comment -

          Ah, ok, I was afraid you meant building it from scratch, which seemed extremely over the top to me. =)

          Show
          Andrew Bayer added a comment - Ah, ok, I was afraid you meant building it from scratch, which seemed extremely over the top to me. =)
          Hide
          Konstantin Boudnik added a comment -

          Well, not exactly 'to build': just get the binary archive, extract and package a couple of files from it. But as the result we'll have a full control of the environment across all the OS spectra.

          Show
          Konstantin Boudnik added a comment - Well, not exactly 'to build': just get the binary archive, extract and package a couple of files from it. But as the result we'll have a full control of the environment across all the OS spectra.
          Hide
          Andrew Bayer added a comment -

          Well, there is a cost if you need to build/package it yourself, isn't there?

          Show
          Andrew Bayer added a comment - Well, there is a cost if you need to build/package it yourself, isn't there?
          Hide
          Konstantin Boudnik added a comment -

          It isn't about features, but bug fixes and performance. Besides, why sticking with odd-years old stuff if newer is available at no cost?

          Show
          Konstantin Boudnik added a comment - It isn't about features, but bug fixes and performance. Besides, why sticking with odd-years old stuff if newer is available at no cost?
          Hide
          Andrew Bayer added a comment -

          Do you really need 2.x features for the script in BIGTOP-952? That doesn't seem likely to me.

          Show
          Andrew Bayer added a comment - Do you really need 2.x features for the script in BIGTOP-952 ? That doesn't seem likely to me.
          Hide
          Konstantin Boudnik added a comment -

          With the above consideration, I think the better idea would be to actually build groovy as the part of our own stack and let it be installed along with the rest of the software. We don't need the full groovy environment - just runtime, essentially, e.g. groovy-<version>.jar; re-distribution will guarantee that we use the environment that we can rely upon, etc.

          Show
          Konstantin Boudnik added a comment - With the above consideration, I think the better idea would be to actually build groovy as the part of our own stack and let it be installed along with the rest of the software. We don't need the full groovy environment - just runtime, essentially, e.g. groovy-<version>.jar; re-distribution will guarantee that we use the environment that we can rely upon, etc.
          Hide
          Konstantin Boudnik added a comment -

          The problem though, is that most of packages for CENTOS and alike are real old: in the hood of Groovy 1.8, where the latest is 2.2 or even later.

          Show
          Konstantin Boudnik added a comment - The problem though, is that most of packages for CENTOS and alike are real old: in the hood of Groovy 1.8, where the latest is 2.2 or even later.
          Hide
          Konstantin Boudnik added a comment -

          The approach I will take is to essentially have a meta-package that will depend on whatever Groovy's supported and provided package out there...

          Show
          Konstantin Boudnik added a comment - The approach I will take is to essentially have a meta-package that will depend on whatever Groovy's supported and provided package out there...
          Konstantin Boudnik made changes -
          Assignee Roman Shaposhnik [ rvs ] Konstantin Boudnik [ cos ]
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Link This issue blocks BIGTOP-952 [ BIGTOP-952 ]
          Roman Shaposhnik created issue -

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development