Hive
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7717

Add .q tests coverage for "union all" [Spark Branch]

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: spark-branch
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      Add automation test coverage for "union all", by searching through the q-tests in "ql/src/test/queries/clientpositive/" for union tests (like union*.q) and verifying/enabling them on spark.

      Steps to do:

      1. Enable a qtest <q-test-name>.q in itests/src/test/resources/testconfiguration.properties by adding the .q test files to spark.query.files.
      2. Run mvn test -Dtest=TestSparkCliDriver -Dqfile=<q-test-name>.q -Dtest.output.overwrite=true -Phadoop-2 to generate the output (located in ql/src/test/results/clientpositive/spark). File will be called <q-test-name>.q.out.
      3. Check the generated output is good by verifying the results. For comparison, check the MR version in ql/src/test/results/clientpositive/<q-test-name>.q.out. The reason its separate is because the explain plan outputs are different for Spark/MR.
      4. Checkin the modification to testconfiguration.properties, and the generated q.out file as well. You only have to generate the output once.

      1. HIVE-7717.3-spark.patch
        304 kB
        Na Yang
      2. HIVE-7717.2-spark.patch
        303 kB
        Na Yang
      3. HIVE-7717.1-spark.patch
        303 kB
        Na Yang

        Issue Links

          Activity

          Hide
          Brock Noland added a comment -

          Thank you very much for your contribution!! I have committed this to spark!

          Show
          Brock Noland added a comment - Thank you very much for your contribution!! I have committed this to spark!
          Hide
          Brock Noland added a comment -

          +1

          those union tests failures are due to HIVE-7786

          Show
          Brock Noland added a comment - +1 those union tests failures are due to HIVE-7786
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12662869/HIVE-7717.3-spark.patch

          ERROR: -1 due to 10 failed/errored test(s), 5958 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union2
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union5
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union7
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union8
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union9
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/63/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/63/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-63/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 10 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12662869

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662869/HIVE-7717.3-spark.patch ERROR: -1 due to 10 failed/errored test(s), 5958 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union2 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union5 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union7 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union8 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union9 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/63/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/63/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-63/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed This message is automatically generated. ATTACHMENT ID: 12662869
          Hide
          Na Yang added a comment -

          Thank you Brock! I attached a new patch based on the latest HEAD.

          Show
          Na Yang added a comment - Thank you Brock! I attached a new patch based on the latest HEAD.
          Hide
          Brock Noland added a comment -

          Also FYI I committed HIVE-7781 so you'll need to pull the latest HEAD.

          Show
          Brock Noland added a comment - Also FYI I committed HIVE-7781 so you'll need to pull the latest HEAD.
          Hide
          Brock Noland added a comment -

          I merged HIVE-7769 into the branch!

          Show
          Brock Noland added a comment - I merged HIVE-7769 into the branch!
          Hide
          Brock Noland added a comment -

          Ok sounds good, please let me know when the patch is up on a separate JIRA for trunk!

          FYI I noticed that Tez tests union2-9 most of which already have a sort:

          https://github.com/apache/hive/blob/trunk/itests/src/test/resources/testconfiguration.properties#L112

          Show
          Brock Noland added a comment - Ok sounds good, please let me know when the patch is up on a separate JIRA for trunk! FYI I noticed that Tez tests union2-9 most of which already have a sort: https://github.com/apache/hive/blob/trunk/itests/src/test/resources/testconfiguration.properties#L112
          Hide
          Na Yang added a comment -

          Hi Brock,

          By looking at those test cases, most of them do not have "order by" following the "union all" operator. For the test cases which pass this time might break in another run. I think we can hold this patch until the .q files are updated in trunk and merged to spark branch. Then I will regenerate the output files from the new .q files with sort enabled. What do you think?

          Thanks,
          Na

          Show
          Na Yang added a comment - Hi Brock, By looking at those test cases, most of them do not have "order by" following the "union all" operator. For the test cases which pass this time might break in another run. I think we can hold this patch until the .q files are updated in trunk and merged to spark branch. Then I will regenerate the output files from the new .q files with sort enabled. What do you think? Thanks, Na
          Hide
          Brock Noland added a comment -

          Gotcha,

          Based on your runs...if the order is not deterministic we can either add an ORDER BY to the query or add the following to the top of the q file:

          -- SORT_BEFORE_DIFF
          

          in either case we'd probably want to make the change on trunk and then merge to our branch since the MR outputs would also need to be updated. We'd probably want to remove those tests from this change and add them in a follow-up.

          Show
          Brock Noland added a comment - Gotcha, Based on your runs...if the order is not deterministic we can either add an ORDER BY to the query or add the following to the top of the q file: -- SORT_BEFORE_DIFF in either case we'd probably want to make the change on trunk and then merge to our branch since the MR outputs would also need to be updated. We'd probably want to remove those tests from this change and add them in a follow-up.
          Hide
          Na Yang added a comment -

          Hi Brock,

          Those tests passed locally. I re-generated some of the test results and found out that the explain plans are the same between test runs, but the result data have different order although the same number of rows are returned. Let me re-ran those tests and upload a new patch.

          Thanks,
          Na

          Show
          Na Yang added a comment - Hi Brock, Those tests passed locally. I re-generated some of the test results and found out that the explain plans are the same between test runs, but the result data have different order although the same number of rows are returned. Let me re-ran those tests and upload a new patch. Thanks, Na
          Hide
          Brock Noland added a comment -

          Hi Na,

          I saw some tests failed. Did those pass locally or are they caused by some feature we don't yet support?

          Also since the patch likely needs an update I committed HIVE-7755 which will cause you a small rebase in testconfiguration.properties.

          Cheers!
          Brock

          Show
          Brock Noland added a comment - Hi Na, I saw some tests failed. Did those pass locally or are they caused by some feature we don't yet support? Also since the patch likely needs an update I committed HIVE-7755 which will cause you a small rebase in testconfiguration.properties. Cheers! Brock
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12662567/HIVE-7717.2-spark.patch

          ERROR: -1 due to 8 failed/errored test(s), 5942 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union14
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union15
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_ppr
          org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/58/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/58/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-58/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 8 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12662567

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662567/HIVE-7717.2-spark.patch ERROR: -1 due to 8 failed/errored test(s), 5942 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union14 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_ppr org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/58/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/58/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-58/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12662567
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12662533/HIVE-7717.1-spark.patch

          ERROR: -1 due to 23 failed/errored test(s), 5942 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union11
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union14
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union15
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union25
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union7
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_null
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_ppr
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/57/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/57/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-57/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 23 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12662533

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662533/HIVE-7717.1-spark.patch ERROR: -1 due to 23 failed/errored test(s), 5942 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union11 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union14 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union25 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union7 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_null org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_ppr Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/57/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/57/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-57/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 23 tests failed This message is automatically generated. ATTACHMENT ID: 12662533
          Hide
          Na Yang added a comment -

          Brock, thank you for informing me about the change. Let me rerun the tests and generate the result file upon the latest code.

          Show
          Na Yang added a comment - Brock, thank you for informing me about the change. Let me rerun the tests and generate the result file upon the latest code.
          Hide
          Brock Noland added a comment -

          Thanks Na! I have just committed HIVE-7528 which might change the explain plan slightly.

          Show
          Brock Noland added a comment - Thanks Na! I have just committed HIVE-7528 which might change the explain plan slightly.
          Hide
          Na Yang added a comment -

          enabled some .q test for union all

          Show
          Na Yang added a comment - enabled some .q test for union all
          Hide
          Brock Noland added a comment -

          FYI -Dqfile= is not usable until HIVE-7739 is resolved. The testconfigration.properties file can be used.

          Show
          Brock Noland added a comment - FYI -Dqfile= is not usable until HIVE-7739 is resolved. The testconfigration.properties file can be used.
          Hide
          Brock Noland added a comment -

          Thanks Na!!

          Show
          Brock Noland added a comment - Thanks Na!!

            People

            • Assignee:
              Na Yang
              Reporter:
              Na Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development