Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0, 1.5.0
    • Component/s: build
    • Labels:
      None

      Description

      This ticket is for improving Accumulo ease-of-installation by adding built-in support for debian packing.

      Debian pkg support requires:
      1. Creating the deb package lifecycle hook scripts, such as 'preinst' and 'postinst'
      2. Creating an init.d script (one of these already exists in bin/)
      3. Integrating deb construction into the build lifecycle. There are some mvn dpkg plugins, but Im not sure how well these work. It might just be easier to run the deb construction as a delegated ant task (via the maven's antrun plugin)

      Outline of required files: Putting all related packing files under src/packages, which is consistent with other Apache projects, such as zookeeper.
      The following dpkg hooks go in src/packages/deb/accumulo.control/:
      conffile, control, postinst, postrm, preinst, prerm
      An init.d script (which I think can just be a copy of bin/etc_initd_accumulo) goes in src/packages/init.d:

      1. ACCUMULO-19_v3.patch
        3 kB
        Scott Kuehn
      2. ACCUMULO-19_v2.patch
        5 kB
        Scott Kuehn
      3. ACCUMULO-19.patch
        3 kB
        Scott Kuehn

        Activity

        Hide
        Scott Kuehn added a comment -

        All – I've started working on this, and am happy to contribute the results of my efforts. Please think it over and let me know.

        Show
        Scott Kuehn added a comment - All – I've started working on this, and am happy to contribute the results of my efforts. Please think it over and let me know.
        Hide
        Billie Rinaldi added a comment -

        Sounds good to me.

        Show
        Billie Rinaldi added a comment - Sounds good to me.
        Hide
        Joey Echeverria added a comment -

        You might also consider contributing this to BigTop to ease integration with the rest of the Hadoop stack.

        Show
        Joey Echeverria added a comment - You might also consider contributing this to BigTop to ease integration with the rest of the Hadoop stack.
        Hide
        Scott Kuehn added a comment -

        Quick status update, FYI –
        I've evaluated these options for integrating debian pkg support:

        • maven-deb plugin: incomplete, and only supports jar projects
        • maven-unix plugin: can not be run as a second-level mvn goal, so the project pom would have to be refactored
        • running ant-plugin, which contains a deb target. This is the way zookeeper's deb and rpm packaging work, but I only want to wrap another build system as a last ditch effort.

        I'm now investigating a newer maven-debian plugin called jdeb: https://github.com/tcurdt/jdeb/blob/master/docs/maven.md
        and it looks promising. My current issue is figuring out how to run the plugin at the end of the packaging phase b/c my current config is executing the jdeb target prior to packaging the main module.

        This ticket also relates to bigger issue of organizing distribution packages, and files related to run-time, such as everything in bin. We should continue considering the addition of a top level distribution package.

        I'm hoping to find some time to pump something out over the next few days.

        Show
        Scott Kuehn added a comment - Quick status update, FYI – I've evaluated these options for integrating debian pkg support: maven-deb plugin: incomplete, and only supports jar projects maven-unix plugin: can not be run as a second-level mvn goal, so the project pom would have to be refactored running ant-plugin, which contains a deb target. This is the way zookeeper's deb and rpm packaging work, but I only want to wrap another build system as a last ditch effort. I'm now investigating a newer maven-debian plugin called jdeb: https://github.com/tcurdt/jdeb/blob/master/docs/maven.md and it looks promising. My current issue is figuring out how to run the plugin at the end of the packaging phase b/c my current config is executing the jdeb target prior to packaging the main module. This ticket also relates to bigger issue of organizing distribution packages, and files related to run-time, such as everything in bin. We should continue considering the addition of a top level distribution package. I'm hoping to find some time to pump something out over the next few days.
        Hide
        Scott Kuehn added a comment -

        @Joey
        Bigtop looks cool- I'll check it out. This would be worthy of a new ticket.

        Show
        Scott Kuehn added a comment - @Joey Bigtop looks cool- I'll check it out. This would be worthy of a new ticket.
        Show
        Benson Margulies added a comment - https://github.com/tcurdt/jdeb/blob/master/docs/maven.md
        Hide
        Bruno Mahé added a comment -

        @Scott
        I can also help integrating your packaging efforts into Bigtop.

        Note: I am a Bigtop commiter

        Show
        Bruno Mahé added a comment - @Scott I can also help integrating your packaging efforts into Bigtop. Note: I am a Bigtop commiter
        Hide
        Scott Kuehn added a comment -

        Initial patch for debian packaging support is attached. To test this out, run 'mvn -N jdeb:jdeb'. A deb file will be generated in target/, and you can use the standard dpkg-deb commands to inspect the archive.

        A few notes:

        • I mirrored the RPM packaging config as much as possible. As such, the debian package should have the same files and metadata as the rpm package
        • Presently, a generated 'deb' package will not have the typical debian control scripts that set up user accounts, init the db, etc.... I just wanted to show a preview of what a basic deb bundling would look like, and I can add control scripts if this patch is determined to be useful.
        • I didn't bind the goal to a particular build phase b/c executing this goal seems like a rare/one-off operation. Note that the rpm plugin is configured in a similar fashion.
        • You should run 'mvn package' prior to invoking the jdeb goal. I realize this is clunky, and I'm working to get a patch for jdeb that will provide cleaner support to aggregate mvn projects like accumulo. Another option, which might be better in the long run, is to refactor accumulo build structure to have a distribution module that will depend on the other modules. assembly, RPM, Debian, and other distro-related concerns could all be encapsulated in this module.

        Thanks for your patience w/this patch, I haven't had much free time lately.

        Show
        Scott Kuehn added a comment - Initial patch for debian packaging support is attached. To test this out, run 'mvn -N jdeb:jdeb'. A deb file will be generated in target/, and you can use the standard dpkg-deb commands to inspect the archive. A few notes: I mirrored the RPM packaging config as much as possible. As such, the debian package should have the same files and metadata as the rpm package Presently, a generated 'deb' package will not have the typical debian control scripts that set up user accounts, init the db, etc.... I just wanted to show a preview of what a basic deb bundling would look like, and I can add control scripts if this patch is determined to be useful. I didn't bind the goal to a particular build phase b/c executing this goal seems like a rare/one-off operation. Note that the rpm plugin is configured in a similar fashion. You should run 'mvn package' prior to invoking the jdeb goal. I realize this is clunky, and I'm working to get a patch for jdeb that will provide cleaner support to aggregate mvn projects like accumulo. Another option, which might be better in the long run, is to refactor accumulo build structure to have a distribution module that will depend on the other modules. assembly, RPM, Debian, and other distro-related concerns could all be encapsulated in this module. Thanks for your patience w/this patch, I haven't had much free time lately.
        Hide
        Eric Newton added a comment -

        Hey Scott,

        I'll try to take a look tomorrow. My home system is Debian (Ubuntu). I know nothing about Debian packaging, though. I'm still trying to learn about the things that cloudera has done for hadoop/hbase packaging and how we might integrate better.

        Thanks!

        -Eric

        Show
        Eric Newton added a comment - Hey Scott, I'll try to take a look tomorrow. My home system is Debian (Ubuntu). I know nothing about Debian packaging, though. I'm still trying to learn about the things that cloudera has done for hadoop/hbase packaging and how we might integrate better. Thanks! -Eric
        Hide
        Scott Kuehn added a comment -

        Sounds good. You might find the 'dpkg-deb' command helpful, and it should be installed by default on your ubuntu system. You can run with '-x' to extract the .deb archive, and '-I' to inspect the pkg meta config.

        Show
        Scott Kuehn added a comment - Sounds good. You might find the 'dpkg-deb' command helpful, and it should be installed by default on your ubuntu system. You can run with '-x' to extract the .deb archive, and '-I' to inspect the pkg meta config.
        Hide
        Scott Kuehn added a comment -

        Quick update: my jdeb patch was accepted, so the next version of mvn-jdeb plugin will work more smoothly with Accumulo. Essentially, the jdeb:jdeb goal will only run once in the lifecycle, rather than once-per submodule.

        Show
        Scott Kuehn added a comment - Quick update: my jdeb patch was accepted, so the next version of mvn-jdeb plugin will work more smoothly with Accumulo. Essentially, the jdeb:jdeb goal will only run once in the lifecycle, rather than once-per submodule.
        Hide
        Scott Kuehn added a comment -

        Refactored this patch by moving deb building mvn directives to a deb profile in src/assembly/pom.xml. The generated deb archive will contain the same files as the RPM archive.

        Run 'mvn package -P deb' to build the deb package.

        Install the deb package on a debian-based system, but running 'dpkg --install <accumulo.deb>' or 'dpkg --unpack <accumulo.deb>

        Show
        Scott Kuehn added a comment - Refactored this patch by moving deb building mvn directives to a deb profile in src/assembly/pom.xml. The generated deb archive will contain the same files as the RPM archive. Run 'mvn package -P deb' to build the deb package. Install the deb package on a debian-based system, but running 'dpkg --install <accumulo.deb>' or 'dpkg --unpack <accumulo.deb>
        Hide
        Eric Newton added a comment -

        Give me a couple of days to go over this. I would really like this to be part of the 1.4 release.

        Expect emails (off ticket) so I can ask stupid questions.

        Show
        Eric Newton added a comment - Give me a couple of days to go over this. I would really like this to be part of the 1.4 release. Expect emails (off ticket) so I can ask stupid questions.
        Hide
        jv added a comment -

        The latter patch would only work if the maven changes I've been making in 1.5 are backported. Take that s you will.

        Show
        jv added a comment - The latter patch would only work if the maven changes I've been making in 1.5 are backported. Take that s you will.
        Hide
        Scott Kuehn added a comment -

        @John good catch. this ticket is marked for 1.4, and the original patch works with version 1.4. The *_v2 patch is for 1.5.
        Not sure what the protocol is for an issue that has patches for two branches. Regardless, both are functional, and have been tested with their respective versions.

        For ACCUMULO-19_v2.patch on v1.5, run: mvn package -P deb
        For ACCUMULO-19.patch on v1.4, run: mvn package && mvn -N jdeb:jdeb

        Show
        Scott Kuehn added a comment - @John good catch. this ticket is marked for 1.4, and the original patch works with version 1.4. The *_v2 patch is for 1.5. Not sure what the protocol is for an issue that has patches for two branches. Regardless, both are functional, and have been tested with their respective versions. For ACCUMULO-19 _v2.patch on v1.5, run: mvn package -P deb For ACCUMULO-19 .patch on v1.4, run: mvn package && mvn -N jdeb:jdeb
        Hide
        jv added a comment -

        we've been discussing backporting the maven changes in trunk to 1.4, so we may just need the one patch

        Show
        jv added a comment - we've been discussing backporting the maven changes in trunk to 1.4, so we may just need the one patch
        Hide
        Eric Newton added a comment -

        Thanks Scott,

        I patched the 1.4 branch, and was able to build a .deb and install it on my Ubuntu laptop. I had to install Cloudera cdh3u2 to satisfy the "hadoop" dependency.

        Shouldn't the package depend on default-jre instead of jdk? Is this a difference between debian and ubuntu?

        I naively tried to run the resulting install, and as expected, it failed horribly. I created ACCUMULO-210 to build on this package for a better user install experience. If you wanted to take on this ticket, that would be great!

        Show
        Eric Newton added a comment - Thanks Scott, I patched the 1.4 branch, and was able to build a .deb and install it on my Ubuntu laptop. I had to install Cloudera cdh3u2 to satisfy the "hadoop" dependency. Shouldn't the package depend on default-jre instead of jdk? Is this a difference between debian and ubuntu? I naively tried to run the resulting install, and as expected, it failed horribly. I created ACCUMULO-210 to build on this package for a better user install experience. If you wanted to take on this ticket, that would be great!
        Hide
        Eric Newton added a comment -

        Thanks Scott,

        I patched the 1.4 branch, and was able to build a .deb and install it on my Ubuntu laptop. I had to install Cloudera cdh3u2 to satisfy the "hadoop" dependency.

        Shouldn't the package depend on default-jre instead of jdk? Is this a difference between debian and ubuntu?

        I naively tried to run the resulting install, and as expected, it failed horribly. I created ACCUMULO-210 to build on this package for a better user install experience. If you wanted to take on this ticket, that would be great!

        Show
        Eric Newton added a comment - Thanks Scott, I patched the 1.4 branch, and was able to build a .deb and install it on my Ubuntu laptop. I had to install Cloudera cdh3u2 to satisfy the "hadoop" dependency. Shouldn't the package depend on default-jre instead of jdk? Is this a difference between debian and ubuntu? I naively tried to run the resulting install, and as expected, it failed horribly. I created ACCUMULO-210 to build on this package for a better user install experience. If you wanted to take on this ticket, that would be great!
        Hide
        Scott Kuehn added a comment -

        @Eric: default-jre-headless is the correct dependency. Debian and Ubuntu will use this 'default' dependency to point to the correct jre package for the particular distro. I uploaded a patch with this change.

        +1 on ACCUMULO-210: a deb/rpm 'out-of-the-box' running accumulo is a great (and obtainable) goal.

        Show
        Scott Kuehn added a comment - @Eric: default-jre-headless is the correct dependency. Debian and Ubuntu will use this 'default' dependency to point to the correct jre package for the particular distro. I uploaded a patch with this change. +1 on ACCUMULO-210 : a deb/rpm 'out-of-the-box' running accumulo is a great (and obtainable) goal.
        Hide
        jv added a comment -

        I am going to make an effort to get these patches rolled in to both 1.4 and 1.5 so we have debian support in place.

        Show
        jv added a comment - I am going to make an effort to get these patches rolled in to both 1.4 and 1.5 so we have debian support in place.
        Hide
        jv added a comment -

        Applied a modified version of the v3 patch to the 1.4 branch. I need to make an update to change it's destination directory. I want to make it more in line with the other Bigtop projects and have it install to the proper directories.

        Show
        jv added a comment - Applied a modified version of the v3 patch to the 1.4 branch. I need to make an update to change it's destination directory. I want to make it more in line with the other Bigtop projects and have it install to the proper directories.
        Hide
        jv added a comment -

        So, in order to do a proper setup, we should segregate the configuration from the main installation. This means a change in our scripts. I'm proposing half-assing it for 1.4, and then for 1.5 we should have a full, proper installation. This means setting up the config files in /etc, appropriate locations for the walogs and logs. Thoughts?

        Show
        jv added a comment - So, in order to do a proper setup, we should segregate the configuration from the main installation. This means a change in our scripts. I'm proposing half-assing it for 1.4, and then for 1.5 we should have a full, proper installation. This means setting up the config files in /etc, appropriate locations for the walogs and logs. Thoughts?
        Hide
        jv added a comment -

        I'm marking this as resolved. Any bugs/problems/features should be new tickets. ACCUMULO-210 is the ticket for role based packages.

        Show
        jv added a comment - I'm marking this as resolved. Any bugs/problems/features should be new tickets. ACCUMULO-210 is the ticket for role based packages.
        Hide
        Eric Newton added a comment -

        start script still broken

        $ ./bin/start-all.sh 
        Accumulo 1.4.0-incubating-SNAPSHOT requires Hadoop version 0.20.x
        
        
        Show
        Eric Newton added a comment - start script still broken $ ./bin/start-all.sh Accumulo 1.4.0-incubating-SNAPSHOT requires Hadoop version 0.20.x
        Hide
        jv added a comment -

        Unbroke things for standard packaging, I'm not certain we can get the /usr/bin/accumulo sorted out without refactoring the scripts. Spun it off into ACCUMULO-432

        Show
        jv added a comment - Unbroke things for standard packaging, I'm not certain we can get the /usr/bin/accumulo sorted out without refactoring the scripts. Spun it off into ACCUMULO-432

          People

          • Assignee:
            Unassigned
            Reporter:
            Scott Kuehn
          • Votes:
            3 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development