Hive
  1. Hive
  2. HIVE-6946

Make it easier to run WebHCat e2e tests

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.14.0
    • Component/s: WebHCat
    • Labels:
      None

      Description

      Right now hcatalog/src/test/e2e/templeton/README.txt explains the steps to set up WebHCat e2e tests but it's cumbersome and error prone. Need to make some improvements here.

      The high level goal here is to simplify code-compile-test loop for WebHCat.

      NO PRECOMMIT TESTS

      1. HIVE-6946.patch
        41 kB
        Eugene Koifman
      2. HIVE-6946.2.patch
        37 kB
        Eugene Koifman
      3. HIVE-6946.3.patch
        34 kB
        Eugene Koifman

        Activity

        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
        Hide
        Thejas M Nair added a comment -

        Patch committed to trunk.
        Thanks for the contribution Eugene!

        Show
        Thejas M Nair added a comment - Patch committed to trunk. Thanks for the contribution Eugene!
        Hide
        Thejas M Nair added a comment -

        +1

        Show
        Thejas M Nair added a comment - +1
        Hide
        Eugene Koifman added a comment -

        done

        Show
        Eugene Koifman added a comment - done
        Hide
        Thejas M Nair added a comment -

        Looks great! Thanks for making webhcat e2e tests easier to run!
        One minor comment, i think the commented out derby config parameters can be removed from the hive-site.mssql.xml .

        Show
        Thejas M Nair added a comment - Looks great! Thanks for making webhcat e2e tests easier to run! One minor comment, i think the commented out derby config parameters can be removed from the hive-site.mssql.xml .
        Hide
        Eugene Koifman added a comment -

        Attached patch contains a few customizable scripts and template config files.

        After setting up a Hadoop cluster, defining a few variables in env.sh and ensuring 3rd party jars (such as proper version of Pig) is available, "restart_hive_redeploy_artifacts.sh" can be used to redeploy all necessary artifacts to the cluster and restart Metastore and WebHcat services so that tests can be run in a clean environment.

        e2e/templeton/deployers/README.txt has more details.

        Show
        Eugene Koifman added a comment - Attached patch contains a few customizable scripts and template config files. After setting up a Hadoop cluster, defining a few variables in env.sh and ensuring 3rd party jars (such as proper version of Pig) is available, "restart_hive_redeploy_artifacts.sh" can be used to redeploy all necessary artifacts to the cluster and restart Metastore and WebHcat services so that tests can be run in a clean environment. e2e/templeton/deployers/README.txt has more details.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12641165/HIVE-6946.patch

        ERROR: -1 due to 43 failed/errored test(s), 5417 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
        org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/6/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/6/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 43 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12641165

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12641165/HIVE-6946.patch ERROR: -1 due to 43 failed/errored test(s), 5417 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/6/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/6/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 43 tests failed This message is automatically generated. ATTACHMENT ID: 12641165
        Hide
        Eugene Koifman added a comment -

        hcatalog/src/test/e2e/templeton/deployers/README.txt in the attached patch explains in detail what's added in the patch

        Show
        Eugene Koifman added a comment - hcatalog/src/test/e2e/templeton/deployers/README.txt in the attached patch explains in detail what's added in the patch

          People

          • Assignee:
            Eugene Koifman
            Reporter:
            Eugene Koifman
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development