Hive
  1. Hive
  2. HIVE-4739

Create public parallel test environment

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Issue Links

        Activity

        Hide
        Brock Noland added a comment -

        As I see it there are three phases:

        1) Executing the post-commit tests on the parallel infrastructure
        2) Providing a manual patch test build
        3) Implementing automated patch testing

        I am interesting this work.

        Show
        Brock Noland added a comment - As I see it there are three phases: 1) Executing the post-commit tests on the parallel infrastructure 2) Providing a manual patch test build 3) Implementing automated patch testing I am interesting this work.
        Hide
        Brock Noland added a comment -

        My employer, Cloudera, has agreed to sponsor a public/open test cluster for the Hive project. The current thought is to run the ptest framework using Apache JClouds (incubating) to spin up "spot instances" via Jenkins for each run. I will be working through setting this up. Just to be clear, once setup it will be available for Hive community use.

        If anyone else is interested in helping to sponsor this project, just comment here and we can work out the details.

        Show
        Brock Noland added a comment - My employer, Cloudera, has agreed to sponsor a public/open test cluster for the Hive project. The current thought is to run the ptest framework using Apache JClouds (incubating) to spin up "spot instances" via Jenkins for each run. I will be working through setting this up. Just to be clear, once setup it will be available for Hive community use. If anyone else is interested in helping to sponsor this project, just comment here and we can work out the details.
        Hide
        Edward Capriolo added a comment -

        +1.

        My dream is to only have to do code reviews, that testing patches would be completely automated. It will take a lot of burden off the committers and make the project more fair and a lower barrier for entry for newer people.

        I am interested to help sponsor.

        There is an important point here. By OPEN the platform has to be OPEN. It can not be open like fabricator/arc, when that process is broken there is no access and no path for support. It needs to be a service running somewhere that any hive committer can access and administer.

        Show
        Edward Capriolo added a comment - +1. My dream is to only have to do code reviews, that testing patches would be completely automated. It will take a lot of burden off the committers and make the project more fair and a lower barrier for entry for newer people. I am interested to help sponsor. There is an important point here. By OPEN the platform has to be OPEN. It can not be open like fabricator/arc, when that process is broken there is no access and no path for support. It needs to be a service running somewhere that any hive committer can access and administer.
        Hide
        Brock Noland added a comment -

        Hi,

        We have the same dream!

        Regarding sponsorship, I am working on a POC with spot instances. If spot instances work there are two items of note:

        1) Sponsorship will be more difficult since they'll be created on demand. I think we'd have to managed multiple EC2 Api keys.
        2) It looks like it will be very, very cheap.

        Regarding "OPEN" - I agree. If the instances are created on-demand, the only item I think we'll need to figure out is how to store the EC2 API key. I am not too versed on EC2 API keys but I am hoping we can create a key for this specific application.

        Brock

        Show
        Brock Noland added a comment - Hi, We have the same dream! Regarding sponsorship, I am working on a POC with spot instances. If spot instances work there are two items of note: 1) Sponsorship will be more difficult since they'll be created on demand. I think we'd have to managed multiple EC2 Api keys. 2) It looks like it will be very, very cheap. Regarding "OPEN" - I agree. If the instances are created on-demand, the only item I think we'll need to figure out is how to store the EC2 API key. I am not too versed on EC2 API keys but I am hoping we can create a key for this specific application. Brock
        Hide
        Edward Capriolo added a comment -

        Is the concept that we can simply send an SVN URL, and a patch URL to a "service" and it runs svn co ; ant test ; ant testreport?

        Show
        Edward Capriolo added a comment - Is the concept that we can simply send an SVN URL, and a patch URL to a "service" and it runs svn co ; ant test ; ant testreport?
        Hide
        Brock Noland added a comment -

        I hope to use Jenkins parameterized builds in terms of the end user interface. Basically you'd go the "Apache-Hive-Trunk-Patch-Build" and then paste in the URL of the patch or upload the patch file. In order to implement this I think we'd need some kind of service simply to keep the EC2 API key semi-private. That is putting it on the Jenkins build slaves probably isn't the best idea.

        Additionally if we were going to allow contributors to test patches this way a service might be required...

        Brock

        Show
        Brock Noland added a comment - I hope to use Jenkins parameterized builds in terms of the end user interface. Basically you'd go the "Apache-Hive-Trunk-Patch-Build" and then paste in the URL of the patch or upload the patch file. In order to implement this I think we'd need some kind of service simply to keep the EC2 API key semi-private. That is putting it on the Jenkins build slaves probably isn't the best idea. Additionally if we were going to allow contributors to test patches this way a service might be required... Brock
        Hide
        Edward Capriolo added a comment -

        I also like the way the Hadoop does it. As an issue is marked 'patch available' it kicks in. Then it reply's back the jira with the test result. Not sure if that is done by an external system querying jira or some type of jira hook.

        Show
        Edward Capriolo added a comment - I also like the way the Hadoop does it. As an issue is marked 'patch available' it kicks in. Then it reply's back the jira with the test result. Not sure if that is done by an external system querying jira or some type of jira hook.
        Hide
        Brock Noland added a comment -

        Edward Capriolo

        OK I think this is ready. To summarize

        I am ready to create accounts for hive committers that are interested. In order to protect the ec2 key, the account will not have full sudo access. However, the account will be able to:

        • Stop/Start/Restart the ptest2 service
        • Push the latest changes to ptest2 from trunk
        • Look at all logs

        How does this sound?

        Show
        Brock Noland added a comment - Edward Capriolo OK I think this is ready. To summarize A dedicated ec2 medium instance runs a webservice which is used to kick of the build The build starts up 8 ec2 spot instances and then runs the build (these stick around for a half hour waiting for another build) It seems to work well, see more details here: https://issues.apache.org/jira/browse/HIVE-4675?focusedCommentId=13701636&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13701636 This is extremely cheap and handling multiple ec2 keys would be complex. Therefore let's just charge this to my employer for now and then later we can add multiple ec2 key support to ptest2. Once HIVE-4815 is resolved I can create the automated "patch available" testing I am ready to create accounts for hive committers that are interested. In order to protect the ec2 key, the account will not have full sudo access. However, the account will be able to: Stop/Start/Restart the ptest2 service Push the latest changes to ptest2 from trunk Look at all logs How does this sound?
        Hide
        Brock Noland added a comment -

        Also in case it wasn't clear above, my plan would be to create individual accounts for hive committers interested in working with this infrastructure.

        Show
        Brock Noland added a comment - Also in case it wasn't clear above, my plan would be to create individual accounts for hive committers interested in working with this infrastructure.
        Hide
        Brock Noland added a comment -

        Hey guys,

        This is pretty much ready to go. Any committers that would like admin access to this infrastructure should send me there uid and public ssh key. This can be shared by encrypting a message (http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x9DC27AFB8E991CC5) with pgp or sending me an email with a service such as https://www.sendinc.com/.

        Brock

        Show
        Brock Noland added a comment - Hey guys, This is pretty much ready to go. Any committers that would like admin access to this infrastructure should send me there uid and public ssh key. This can be shared by encrypting a message ( http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x9DC27AFB8E991CC5 ) with pgp or sending me an email with a service such as https://www.sendinc.com/ . Brock
        Hide
        Brock Noland added a comment -

        uid should be username above.

        Show
        Brock Noland added a comment - uid should be username above.
        Hide
        Brock Noland added a comment -

        Ashutosh Chauhan Carl Steinbach or any other committer did you guys want admin accounts on this testing infrastructure? If so just shoot me an email with the username you'd like and I'll create an account and password for you.

        Show
        Brock Noland added a comment - Ashutosh Chauhan Carl Steinbach or any other committer did you guys want admin accounts on this testing infrastructure? If so just shoot me an email with the username you'd like and I'll create an account and password for you.
        Hide
        Ashutosh Chauhan added a comment -

        Thanks Brock Noland for setting this up. I will shoot you an email.

        Show
        Ashutosh Chauhan added a comment - Thanks Brock Noland for setting this up. I will shoot you an email.
        Hide
        Carl Steinbach added a comment -

        Brock Noland Ditto. Thanks!

        Show
        Carl Steinbach added a comment - Brock Noland Ditto. Thanks!
        Hide
        Brock Noland added a comment -

        I am going to mark this resolved. Two PMC members have accounts and a third has expressed interest. Thanks guys!

        Show
        Brock Noland added a comment - I am going to mark this resolved. Two PMC members have accounts and a third has expressed interest. Thanks guys!

          People

          • Assignee:
            Brock Noland
            Reporter:
            Brock Noland
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development