Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Testing Infrastructure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HIVE-1464 speeded up serial runs somewhat - but looks like it's still too slow. we should use parallel junit or some similar setup to run test queries in parallel. this should be really easy as well need to just use a separate warehouse/metadb and potentiall mapred system dir location.

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Ashutosh Chauhan added a comment -

        Awesome work! Thanks, Marek and everyone else involved for making life better for us : )

        Show
        Ashutosh Chauhan added a comment - Awesome work! Thanks, Marek and everyone else involved for making life better for us : )
        Show
        John Sichi added a comment - https://cwiki.apache.org/confluence/display/Hive/Unit+Test+Parallel+Execution
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1141 (See https://builds.apache.org/job/Hive-trunk-h0.21/1141/)
        Fix executable permissions left out of commit for HIVE-1487.

        jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213433
        Files :

        • /hive/trunk/testutils/ptest/Report.py
        • /hive/trunk/testutils/ptest/hivetest.py
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1141 (See https://builds.apache.org/job/Hive-trunk-h0.21/1141/ ) Fix executable permissions left out of commit for HIVE-1487 . jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213433 Files : /hive/trunk/testutils/ptest/Report.py /hive/trunk/testutils/ptest/hivetest.py
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.23.0 #17 (See https://builds.apache.org/job/Hive-trunk-h0.23.0/17/)
        Fix executable permissions left out of commit for HIVE-1487.

        jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213433
        Files :

        • /hive/trunk/testutils/ptest/Report.py
        • /hive/trunk/testutils/ptest/hivetest.py
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.23.0 #17 (See https://builds.apache.org/job/Hive-trunk-h0.23.0/17/ ) Fix executable permissions left out of commit for HIVE-1487 . jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1213433 Files : /hive/trunk/testutils/ptest/Report.py /hive/trunk/testutils/ptest/hivetest.py
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1137 (See https://builds.apache.org/job/Hive-trunk-h0.21/1137/)
        HIVE-1487 [jira] Add ptest script
        (Marek Sapota via John Sichi)

        Summary: HIVE-1464 speeded up serial runs somewhat - but looks like it's still
        too slow. we should use parallel junit or some similar setup to run test queries
        in parallel. this should be really easy as well need to just use a separate
        warehouse/metadb and potentiall mapred system dir location.

        Test Plan: EMPTY

        Reviewers: JIRA, jsichi

        Reviewed By: jsichi

        CC: mareksapotafb, jsichi

        Differential Revision: 603

        jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212697
        Files :

        • /hive/trunk/testutils/ptest
        • /hive/trunk/testutils/ptest/Buffer.py
        • /hive/trunk/testutils/ptest/Process.py
        • /hive/trunk/testutils/ptest/README
        • /hive/trunk/testutils/ptest/Report.py
        • /hive/trunk/testutils/ptest/Ssh.py
        • /hive/trunk/testutils/ptest/config.py
        • /hive/trunk/testutils/ptest/hivetest.py
        • /hive/trunk/testutils/ptest/templates
        • /hive/trunk/testutils/ptest/templates/Properties.html
        • /hive/trunk/testutils/ptest/templates/TestCase.html
        • /hive/trunk/testutils/ptest/templates/TestRun.html
        • /hive/trunk/testutils/ptest/templates/TestSuite.html
        • /hive/trunk/testutils/ptest/templates/common.css
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1137 (See https://builds.apache.org/job/Hive-trunk-h0.21/1137/ ) HIVE-1487 [jira] Add ptest script (Marek Sapota via John Sichi) Summary: HIVE-1464 speeded up serial runs somewhat - but looks like it's still too slow. we should use parallel junit or some similar setup to run test queries in parallel. this should be really easy as well need to just use a separate warehouse/metadb and potentiall mapred system dir location. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: mareksapotafb, jsichi Differential Revision: 603 jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212697 Files : /hive/trunk/testutils/ptest /hive/trunk/testutils/ptest/Buffer.py /hive/trunk/testutils/ptest/Process.py /hive/trunk/testutils/ptest/README /hive/trunk/testutils/ptest/Report.py /hive/trunk/testutils/ptest/Ssh.py /hive/trunk/testutils/ptest/config.py /hive/trunk/testutils/ptest/hivetest.py /hive/trunk/testutils/ptest/templates /hive/trunk/testutils/ptest/templates/Properties.html /hive/trunk/testutils/ptest/templates/TestCase.html /hive/trunk/testutils/ptest/templates/TestRun.html /hive/trunk/testutils/ptest/templates/TestSuite.html /hive/trunk/testutils/ptest/templates/common.css
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.23.0 #13 (See https://builds.apache.org/job/Hive-trunk-h0.23.0/13/)
        HIVE-1487 [jira] Add ptest script
        (Marek Sapota via John Sichi)

        Summary: HIVE-1464 speeded up serial runs somewhat - but looks like it's still
        too slow. we should use parallel junit or some similar setup to run test queries
        in parallel. this should be really easy as well need to just use a separate
        warehouse/metadb and potentiall mapred system dir location.

        Test Plan: EMPTY

        Reviewers: JIRA, jsichi

        Reviewed By: jsichi

        CC: mareksapotafb, jsichi

        Differential Revision: 603

        jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212697
        Files :

        • /hive/trunk/testutils/ptest
        • /hive/trunk/testutils/ptest/Buffer.py
        • /hive/trunk/testutils/ptest/Process.py
        • /hive/trunk/testutils/ptest/README
        • /hive/trunk/testutils/ptest/Report.py
        • /hive/trunk/testutils/ptest/Ssh.py
        • /hive/trunk/testutils/ptest/config.py
        • /hive/trunk/testutils/ptest/hivetest.py
        • /hive/trunk/testutils/ptest/templates
        • /hive/trunk/testutils/ptest/templates/Properties.html
        • /hive/trunk/testutils/ptest/templates/TestCase.html
        • /hive/trunk/testutils/ptest/templates/TestRun.html
        • /hive/trunk/testutils/ptest/templates/TestSuite.html
        • /hive/trunk/testutils/ptest/templates/common.css
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.23.0 #13 (See https://builds.apache.org/job/Hive-trunk-h0.23.0/13/ ) HIVE-1487 [jira] Add ptest script (Marek Sapota via John Sichi) Summary: HIVE-1464 speeded up serial runs somewhat - but looks like it's still too slow. we should use parallel junit or some similar setup to run test queries in parallel. this should be really easy as well need to just use a separate warehouse/metadb and potentiall mapred system dir location. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: mareksapotafb, jsichi Differential Revision: 603 jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1212697 Files : /hive/trunk/testutils/ptest /hive/trunk/testutils/ptest/Buffer.py /hive/trunk/testutils/ptest/Process.py /hive/trunk/testutils/ptest/README /hive/trunk/testutils/ptest/Report.py /hive/trunk/testutils/ptest/Ssh.py /hive/trunk/testutils/ptest/config.py /hive/trunk/testutils/ptest/hivetest.py /hive/trunk/testutils/ptest/templates /hive/trunk/testutils/ptest/templates/Properties.html /hive/trunk/testutils/ptest/templates/TestCase.html /hive/trunk/testutils/ptest/templates/TestRun.html /hive/trunk/testutils/ptest/templates/TestSuite.html /hive/trunk/testutils/ptest/templates/common.css
        Hide
        Phabricator added a comment -

        mareksapotafb has committed the revision "HIVE-1487 [jira] Add ptest script".

        REVISION DETAIL
        https://reviews.facebook.net/D603

        COMMIT
        https://reviews.facebook.net/rHIVE1212697

        Show
        Phabricator added a comment - mareksapotafb has committed the revision " HIVE-1487 [jira] Add ptest script". REVISION DETAIL https://reviews.facebook.net/D603 COMMIT https://reviews.facebook.net/rHIVE1212697
        Hide
        John Sichi added a comment -

        Also thanks Zheng for the original ptest script, and Charles Chen for hacking up an intermediate version

        Show
        John Sichi added a comment - Also thanks Zheng for the original ptest script, and Charles Chen for hacking up an intermediate version
        Hide
        John Sichi added a comment -

        Committed to trunk. Thanks Marek!

        Show
        John Sichi added a comment - Committed to trunk. Thanks Marek!
        Hide
        Phabricator added a comment -

        jsichi has accepted the revision "HIVE-1487 [jira] Add ptest script".

        +1, committing now. Can you grant rights to ASF?

        REVISION DETAIL
        https://reviews.facebook.net/D603

        Show
        Phabricator added a comment - jsichi has accepted the revision " HIVE-1487 [jira] Add ptest script". +1, committing now. Can you grant rights to ASF? REVISION DETAIL https://reviews.facebook.net/D603
        Hide
        Phabricator added a comment -

        mareksapotafb has commented on the revision "HIVE-1487 [jira] Add ptest script".

        Just use instructions from the wiki, the instance set up in my home directory uses this patch.

        REVISION DETAIL
        https://reviews.facebook.net/D603

        Show
        Phabricator added a comment - mareksapotafb has commented on the revision " HIVE-1487 [jira] Add ptest script". Just use instructions from the wiki, the instance set up in my home directory uses this patch. REVISION DETAIL https://reviews.facebook.net/D603
        Hide
        Phabricator added a comment -

        jsichi has commented on the revision "HIVE-1487 [jira] Add ptest script".

        Do you have this deployed somewhere I can test it?

        REVISION DETAIL
        https://reviews.facebook.net/D603

        Show
        Phabricator added a comment - jsichi has commented on the revision " HIVE-1487 [jira] Add ptest script". Do you have this deployed somewhere I can test it? REVISION DETAIL https://reviews.facebook.net/D603
        Hide
        Phabricator added a comment -

        mareksapotafb updated the revision "HIVE-1487 [jira] Add ptest script".
        Reviewers: JIRA, jsichi

        This should fix the Minimr issues.

        REVISION DETAIL
        https://reviews.facebook.net/D603

        AFFECTED FILES
        testutils/ptest/Buffer.py
        testutils/ptest/Process.py
        testutils/ptest/README
        testutils/ptest/Report.py
        testutils/ptest/Ssh.py
        testutils/ptest/config.py
        testutils/ptest/hivetest.py
        testutils/ptest/templates/Properties.html
        testutils/ptest/templates/TestCase.html
        testutils/ptest/templates/TestRun.html
        testutils/ptest/templates/TestSuite.html
        testutils/ptest/templates/common.css

        Show
        Phabricator added a comment - mareksapotafb updated the revision " HIVE-1487 [jira] Add ptest script". Reviewers: JIRA, jsichi This should fix the Minimr issues. REVISION DETAIL https://reviews.facebook.net/D603 AFFECTED FILES testutils/ptest/Buffer.py testutils/ptest/Process.py testutils/ptest/README testutils/ptest/Report.py testutils/ptest/Ssh.py testutils/ptest/config.py testutils/ptest/hivetest.py testutils/ptest/templates/Properties.html testutils/ptest/templates/TestCase.html testutils/ptest/templates/TestRun.html testutils/ptest/templates/TestSuite.html testutils/ptest/templates/common.css
        Hide
        Phabricator added a comment -

        mareksapotafb updated the revision "HIVE-1487 [jira] Add ptest script".
        Reviewers: JIRA, jsichi

        It should, I made a mistake manually writing a commit message. It should get fixed after this diff.

        REVISION DETAIL
        https://reviews.facebook.net/D603

        AFFECTED FILES
        testutils/ptest/Buffer.py
        testutils/ptest/Process.py
        testutils/ptest/README
        testutils/ptest/Report.py
        testutils/ptest/Ssh.py
        testutils/ptest/config.py
        testutils/ptest/hivetest.py
        testutils/ptest/templates/Properties.html
        testutils/ptest/templates/TestCase.html
        testutils/ptest/templates/TestRun.html
        testutils/ptest/templates/TestSuite.html
        testutils/ptest/templates/common.css

        Show
        Phabricator added a comment - mareksapotafb updated the revision " HIVE-1487 [jira] Add ptest script". Reviewers: JIRA, jsichi It should, I made a mistake manually writing a commit message. It should get fixed after this diff. REVISION DETAIL https://reviews.facebook.net/D603 AFFECTED FILES testutils/ptest/Buffer.py testutils/ptest/Process.py testutils/ptest/README testutils/ptest/Report.py testutils/ptest/Ssh.py testutils/ptest/config.py testutils/ptest/hivetest.py testutils/ptest/templates/Properties.html testutils/ptest/templates/TestCase.html testutils/ptest/templates/TestRun.html testutils/ptest/templates/TestSuite.html testutils/ptest/templates/common.css
        Hide
        Phabricator added a comment -

        jsichi has requested changes to the revision "HIVE-1487 [jira] parallelize test query runs".

        We can commit this once we get to the bottom of minimr failures.

        INLINE COMMENTS
        testutils/ptest/README:48 Typo: loose -> lose
        testutils/ptest/Ssh.py:21 Let's expunge this comment.
        testutils/ptest/config.py:83 Typo: faled -> failed

        REVISION DETAIL
        https://reviews.facebook.net/D603

        Show
        Phabricator added a comment - jsichi has requested changes to the revision " HIVE-1487 [jira] parallelize test query runs". We can commit this once we get to the bottom of minimr failures. INLINE COMMENTS testutils/ptest/README:48 Typo: loose -> lose testutils/ptest/Ssh.py:21 Let's expunge this comment. testutils/ptest/config.py:83 Typo: faled -> failed REVISION DETAIL https://reviews.facebook.net/D603
        Hide
        Phabricator added a comment -

        mareksapotafb has added reviewers to the revision "HIVE-1487 [jira] parallelize test query runs".
        Added Reviewers: jsichi

        REVISION DETAIL
        https://reviews.facebook.net/D603

        Show
        Phabricator added a comment - mareksapotafb has added reviewers to the revision " HIVE-1487 [jira] parallelize test query runs". Added Reviewers: jsichi REVISION DETAIL https://reviews.facebook.net/D603
        Hide
        Phabricator added a comment -

        mareksapotafb requested code review of "HIVE-1487 [jira] parallelize test query runs".
        Reviewers: JIRA

        Add ptest script

        <a href="https://issues.apache.org/jira/browse/HIVE-1464" title="improve test query performance"><del>HIVE-1464</del></a> speeded up serial runs somewhat - but looks like it's still too slow. we should use parallel junit or some similar setup to run test queries in parallel. this should be really easy as well need to just use a separate warehouse/metadb and potentiall mapred system dir location.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D603

        AFFECTED FILES
        testutils/ptest/Buffer.py
        testutils/ptest/Process.py
        testutils/ptest/README
        testutils/ptest/Report.py
        testutils/ptest/Ssh.py
        testutils/ptest/config.py
        testutils/ptest/hivetest.py
        testutils/ptest/templates/Properties.html
        testutils/ptest/templates/TestCase.html
        testutils/ptest/templates/TestRun.html
        testutils/ptest/templates/TestSuite.html
        testutils/ptest/templates/common.css

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/1335/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - mareksapotafb requested code review of " HIVE-1487 [jira] parallelize test query runs". Reviewers: JIRA Add ptest script <a href="https://issues.apache.org/jira/browse/HIVE-1464" title="improve test query performance"><del> HIVE-1464 </del></a> speeded up serial runs somewhat - but looks like it's still too slow. we should use parallel junit or some similar setup to run test queries in parallel. this should be really easy as well need to just use a separate warehouse/metadb and potentiall mapred system dir location. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D603 AFFECTED FILES testutils/ptest/Buffer.py testutils/ptest/Process.py testutils/ptest/README testutils/ptest/Report.py testutils/ptest/Ssh.py testutils/ptest/config.py testutils/ptest/hivetest.py testutils/ptest/templates/Properties.html testutils/ptest/templates/TestCase.html testutils/ptest/templates/TestRun.html testutils/ptest/templates/TestSuite.html testutils/ptest/templates/common.css MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1335/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Joydeep Sen Sarma added a comment -

        yeah - that would be my gut feel too (just ditch junit)

        however - we are going to lose the junit style test outputs etc. long time back Ashish did all the velocity stuff to have junit tests. i don't remember the exact thinking at that time - but a majority of people wanted to use junit.

        threading would actually be good though .. (we have a separate multithreaded test right now that we could happily obsolete)

        Show
        Joydeep Sen Sarma added a comment - yeah - that would be my gut feel too (just ditch junit) however - we are going to lose the junit style test outputs etc. long time back Ashish did all the velocity stuff to have junit tests. i don't remember the exact thinking at that time - but a majority of people wanted to use junit. threading would actually be good though .. (we have a separate multithreaded test right now that we could happily obsolete)
        Hide
        John Sichi added a comment -

        Zheng has a private ptest shell script which takes a different approach: it enumerates all of the .q files, partitions them modulo the degree of parallelism, and then executes them in separate environments as separate processes. One advantage is that there's no cross-talk from threading (although in the long run testing with multithreading for independent tests would be good for coverage).

        I'll send you a pointer.

        Show
        John Sichi added a comment - Zheng has a private ptest shell script which takes a different approach: it enumerates all of the .q files, partitions them modulo the degree of parallelism, and then executes them in separate environments as separate processes. One advantage is that there's no cross-talk from threading (although in the long run testing with multithreading for independent tests would be good for coverage). I'll send you a pointer.
        Hide
        Joydeep Sen Sarma added a comment -

        can people with experience running java tests in parallel comment on this? So far these seem the choices:

        • upgrade to junit4 and use custom runner that runs in parallel. the downside here is that junit does not seem to come with this parallel runner (but there's additional code on the web from the junit authors that does the same)
        • use parallel-junit. this seems the least disruptive - but this seems like an old/dead project
        • use TestNG - this is a replacement for junit that has inbuilt parallel execution support. but we would not be using junit anymore at all.

        any other thoughts on better test setup welcome as well.

        Show
        Joydeep Sen Sarma added a comment - can people with experience running java tests in parallel comment on this? So far these seem the choices: upgrade to junit4 and use custom runner that runs in parallel. the downside here is that junit does not seem to come with this parallel runner (but there's additional code on the web from the junit authors that does the same) use parallel-junit. this seems the least disruptive - but this seems like an old/dead project use TestNG - this is a replacement for junit that has inbuilt parallel execution support. but we would not be using junit anymore at all. any other thoughts on better test setup welcome as well.

          People

          • Assignee:
            Marek Sapota
            Reporter:
            Joydeep Sen Sarma
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development