Details

    • Type: New Feature
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: integration tests, util
    • Labels:
      None

      Description

      In the failsafe docs they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing.

      The folks over in Kiji made this great plugin that lets you do just that! We should see about bringing that under the HBase umbrella proper.

        Activity

        Hide
        jesse_yates Jesse Yates added a comment -

        Ok, so looking through the ip clearance template I think that having WibiData's CLA and all active committers have a signed CLA on record should be enough.

        From our end, we can check the CLA:

        Identify name recorded for software grant: the name of the grant as recorded in the foundation/officers area, in either grants.txt or cclas.txt, so that the grant can be easily identified

        However, we need a Chair/Member of ASF to do the actual work (Andrew Purtell? stack?) The rest of the process we need to follow is outlined in steps 5-8 in the ip clearance template.

        Did as much as I could, which was checking the dependencies and starting the clearance form: https://gist.github.com/jyates/67a54dc8c1f23257c925

        Guess we (HBase) need to have a vote as well to accept the code.

        Show
        jesse_yates Jesse Yates added a comment - Ok, so looking through the ip clearance template I think that having WibiData's CLA and all active committers have a signed CLA on record should be enough. From our end, we can check the CLA: Identify name recorded for software grant: the name of the grant as recorded in the foundation/officers area, in either grants.txt or cclas.txt, so that the grant can be easily identified However, we need a Chair/Member of ASF to do the actual work ( Andrew Purtell ? stack ?) The rest of the process we need to follow is outlined in steps 5-8 in the ip clearance template . Did as much as I could, which was checking the dependencies and starting the clearance form: https://gist.github.com/jyates/67a54dc8c1f23257c925 Guess we (HBase) need to have a vote as well to accept the code.
        Hide
        gwu Garrett Wu added a comment -

        Sorry everyone, this just got lost in mail filters. You just need me to fill out https://www.apache.org/licenses/software-grant.txt? I believe that WibiData already has a https://www.apache.org/licenses/cla-corporate.txt on file – how do I verify?

        Show
        gwu Garrett Wu added a comment - Sorry everyone, this just got lost in mail filters. You just need me to fill out https://www.apache.org/licenses/software-grant.txt? I believe that WibiData already has a https://www.apache.org/licenses/cla-corporate.txt on file – how do I verify?
        Hide
        jesse_yates Jesse Yates added a comment -

        Sad face - any time for this Garrett Wu?

        Show
        jesse_yates Jesse Yates added a comment - Sad face - any time for this Garrett Wu ?
        Hide
        apurtell Andrew Purtell added a comment -

        Haven't heard anything back from Garrett Wu here. Let's table this. Resolving as Later.

        Show
        apurtell Andrew Purtell added a comment - Haven't heard anything back from Garrett Wu here. Let's table this. Resolving as Later.
        Hide
        jesse_yates Jesse Yates added a comment -

        Tarball of hbase-maven-plugin (with the commit ID from whence it came).

        Show
        jesse_yates Jesse Yates added a comment - Tarball of hbase-maven-plugin (with the commit ID from whence it came).
        Hide
        jesse_yates Jesse Yates added a comment -

        Adding support, -Dminicluster.verson or whatever, for launching earlier or later releases when running integration tests, for checking forwards and backwards client compat.

        I think with this, you would actually just specify the version of the plugin to match the version of the minicluster you want to run, since we would release the plugin alongside the rest of the codebase. If we didn't have a minicluster plugin for that version, you are SOL (or we could back port it), but otherwise it will run that version of the HBase code.

        Show
        jesse_yates Jesse Yates added a comment - Adding support, -Dminicluster.verson or whatever, for launching earlier or later releases when running integration tests, for checking forwards and backwards client compat. I think with this, you would actually just specify the version of the plugin to match the version of the minicluster you want to run, since we would release the plugin alongside the rest of the codebase. If we didn't have a minicluster plugin for that version, you are SOL (or we could back port it), but otherwise it will run that version of the HBase code.
        Hide
        jesse_yates Jesse Yates added a comment -

        Ah, ok. Sorry Andrew Purtell, should have RTFS. thanks!

        Show
        jesse_yates Jesse Yates added a comment - Ah, ok. Sorry Andrew Purtell , should have RTFS. thanks!
        Hide
        jesse_yates Jesse Yates added a comment - - edited

        FWIW, spent a little bit bringing the plugin up to trunk (github); well, with just hadoop 2.5.1 support, but not sure how we do version support anyways, so that will do the moment.

        Also, brought up my next improvements to it that allow you to do things like running mvn org.apache.hbase:hbase-maven-plugin:run to bring up (and leave up) mini-cluster

        Show
        jesse_yates Jesse Yates added a comment - - edited FWIW, spent a little bit bringing the plugin up to trunk ( github ); well, with just hadoop 2.5.1 support, but not sure how we do version support anyways, so that will do the moment. Also, brought up my next improvements to it that allow you to do things like running mvn org.apache.hbase:hbase-maven-plugin:run to bring up (and leave up) mini-cluster
        Hide
        apurtell Andrew Purtell added a comment -

        No, we are not bringing it into the incubator.
        http://incubator.apache.org/ip-clearance/

        This form is not for new projects. This is for projects and PMCs that have already been created and are receiving a code donation into an existing codebase. Any code that was developed outside of the ASF SVN repository and our public mailing lists must be processed like this, even if the external developer is already an ASF committer.

        Yes, we need to process the donation through the IP clearance process.

        Show
        apurtell Andrew Purtell added a comment - No, we are not bringing it into the incubator. http://incubator.apache.org/ip-clearance/ This form is not for new projects. This is for projects and PMCs that have already been created and are receiving a code donation into an existing codebase. Any code that was developed outside of the ASF SVN repository and our public mailing lists must be processed like this, even if the external developer is already an ASF committer. Yes, we need to process the donation through the IP clearance process.
        Hide
        jesse_yates Jesse Yates added a comment -

        Do we need to bring it into the incubator? That seems a bit much for what is really not that much code (few hundred lines). I was hoping it would just fall in as a new module in HBase.

        If we need to, then we need to... and I'd be +1 for that too.

        Show
        jesse_yates Jesse Yates added a comment - Do we need to bring it into the incubator? That seems a bit much for what is really not that much code (few hundred lines). I was hoping it would just fall in as a new module in HBase. If we need to, then we need to... and I'd be +1 for that too.
        Hide
        apurtell Andrew Purtell added a comment -

        Anyway, before thinking about proceeding - should we?
        I'd be +1

        Show
        apurtell Andrew Purtell added a comment - Anyway, before thinking about proceeding - should we? I'd be +1
        Hide
        apurtell Andrew Purtell added a comment -

        I find applications in hbase-it interesting:

        • Refactoring hbase-it so we don't deal with local cluster setup and tear down for 'mvn verify', let the plugin deal with it
        • Adding support, -Dminicluster.verson or whatever, for launching earlier or later releases when running integration tests, for checking forwards and backwards client compat.
        • Assist downstreamers who want to build on hbase-it
        Show
        apurtell Andrew Purtell added a comment - I find applications in hbase-it interesting: Refactoring hbase-it so we don't deal with local cluster setup and tear down for 'mvn verify', let the plugin deal with it Adding support, -Dminicluster.verson or whatever, for launching earlier or later releases when running integration tests, for checking forwards and backwards client compat. Assist downstreamers who want to build on hbase-it
        Hide
        apurtell Andrew Purtell added a comment -

        We need to follow this process: http://incubator.apache.org/ip-clearance/ip-clearance-template.html

        Generally, we need: a snapshot of the code (tarball), a Software Grant, and a CLA. The code must have no dependencies on items with licenses incompatible with the ASL. I am a Member of the ASF with Incubator karma so can execute the process assuming we want to do this.

        To move forward, we need a snapshot, and since Garrett Wu has already said offline he can represent the copyright holder and is willing to execute a software grant, let's ask him to draw up a software grant (https://www.apache.org/licenses/software-grant.txt) and CCLA (https://www.apache.org/licenses/cla-corporate.txt), if one for Wibi isn't already on file, and submit them to the Apache Secretary (secretary@apache.org)

        Once we have a snapshot I can open a new clearance form. Once we have acknowledgement from the Secretary that the Grant and CLA are on file, I can commit the snapshot over at the incubator and start a clearance vote. Assuming that passes, I think we should commit the snapshot as-is on a branch, massage it post commit, then call a branch merge vote to get it into master.

        Show
        apurtell Andrew Purtell added a comment - We need to follow this process: http://incubator.apache.org/ip-clearance/ip-clearance-template.html Generally, we need: a snapshot of the code (tarball), a Software Grant, and a CLA. The code must have no dependencies on items with licenses incompatible with the ASL. I am a Member of the ASF with Incubator karma so can execute the process assuming we want to do this. To move forward, we need a snapshot, and since Garrett Wu has already said offline he can represent the copyright holder and is willing to execute a software grant, let's ask him to draw up a software grant ( https://www.apache.org/licenses/software-grant.txt ) and CCLA ( https://www.apache.org/licenses/cla-corporate.txt ), if one for Wibi isn't already on file, and submit them to the Apache Secretary (secretary@apache.org) Once we have a snapshot I can open a new clearance form. Once we have acknowledgement from the Secretary that the Grant and CLA are on file, I can commit the snapshot over at the incubator and start a clearance vote. Assuming that passes, I think we should commit the snapshot as-is on a branch, massage it post commit, then call a branch merge vote to get it into master.
        Hide
        jesse_yates Jesse Yates added a comment -

        Basic Cross version compat testing? So you could test a new version of HBase against the minicluster from the plugin in the last version. At least, that's all I can think of right now.

        But still would dramatically help us in cutting down non-destructive, integration test times. For downstreamers, like Phoenix, they wouldn't need to spin up/down a minicluster for every test (which ends up being a majority of the time spent in the test).

        It just seems like something the hbase project should own too

        Show
        jesse_yates Jesse Yates added a comment - Basic Cross version compat testing? So you could test a new version of HBase against the minicluster from the plugin in the last version. At least, that's all I can think of right now. But still would dramatically help us in cutting down non-destructive, integration test times. For downstreamers, like Phoenix, they wouldn't need to spin up/down a minicluster for every test (which ends up being a majority of the time spent in the test). It just seems like something the hbase project should own too
        Hide
        stack stack added a comment -

        So, folks doing integration tests on a single node would use this? Any other usages?
        St.Ack

        Show
        stack stack added a comment - So, folks doing integration tests on a single node would use this? Any other usages? St.Ack
        Hide
        jesse_yates Jesse Yates added a comment -

        The other reason to bring it into HBase is that the hbase-maven-plugin hasn't seen any activity in the last year or so, whereas I think the HBase folks will give it more love than its seen in a while.

        Talked with Garrett Wu offline (one of the committers there) and seemed receptive to bringing it in.

        My biggest question is around licensing; its already Apache 2.0 licensed, what do we need to do to bring it into HBase (besides fixing it up to work on trunk)? Andrew Purtell stack?

        Show
        jesse_yates Jesse Yates added a comment - The other reason to bring it into HBase is that the hbase-maven-plugin hasn't seen any activity in the last year or so, whereas I think the HBase folks will give it more love than its seen in a while. Talked with Garrett Wu offline (one of the committers there) and seemed receptive to bringing it in. My biggest question is around licensing; its already Apache 2.0 licensed, what do we need to do to bring it into HBase (besides fixing it up to work on trunk)? Andrew Purtell stack ?

          People

          • Assignee:
            Unassigned
            Reporter:
            jesse_yates Jesse Yates
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development