Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3491

Reduce number of serial tests with unique database fixture

    Details

      Description

      We have a significant number of tests that have to run serial due to shared state among parallel tests/dimensions, most prominently transient test databases and tables.

      Michael B recently introduced a new pytest fixture for creating an isolated unique database. The task here is to roll out the new fixture to all existing tests.

      I will add a comment for every commit that relates to fixing these tests. Here is a complete list of tests that could benefit from the new fixture:

      ./tests/data_errors/test_data_errors.py
      ./tests/custom_cluster/test_insert_behaviour.py
      ./tests/custom_cluster/test_hdfs_fd_caching.py
      ./tests/custom_cluster/test_permanent_udfs.py
      ./tests/custom_cluster/test_parquet_max_page_header.py
      ./tests/catalog_service/test_catalog_service_client.py
      ./tests/metadata/test_show_create_table.py
      ./tests/metadata/test_metadata_query_statements.py
      ./tests/metadata/test_last_ddl_time_update.py
      ./tests/metadata/test_hbase_metadata.py
      ./tests/metadata/test_hdfs_permissions.py
      ./tests/metadata/test_compute_stats.py
      ./tests/metadata/test_load.py
      ./tests/metadata/test_ddl.py
      ./tests/metadata/test_explain.py
      ./tests/metadata/test_views_compatibility.py
      ./tests/metadata/test_hidden_files.py
      ./tests/metadata/test_recover_partitions.py
      ./tests/metadata/test_hdfs_encryption.py
      ./tests/metadata/test_stale_metadata.py
      ./tests/metadata/test_col_stats.py
      ./tests/metadata/test_partition_metadata.py
      ./tests/shell/test_shell_commandline.py
      ./tests/shell/test_shell_interactive.py
      ./tests/query_test/test_hdfs_caching.py
      ./tests/query_test/test_insert.py
      ./tests/query_test/test_insert_behaviour.py
      ./tests/query_test/test_hdfs_fd_caching.py
      ./tests/query_test/test_cancellation.py
      ./tests/query_test/test_join_queries.py
      ./tests/query_test/test_partitioning.py
      ./tests/query_test/test_chars.py
      ./tests/query_test/test_insert_parquet.py
      ./tests/query_test/test_udfs.py
      ./tests/query_test/test_nested_types.py
      ./tests/query_test/test_compressed_formats.py
      ./tests/query_test/test_local_fs.py
      ./tests/query_test/test_multiple_filesystems.py
      ./tests/query_test/test_scanners.py
      ./tests/query_test/test_delimited_text.py
      ./tests/stress/test_ddl_stress.py
      ./tests/query_test/test_kudu.py

        Activity

        Hide
        alex.behm Alexander Behm added a comment -

        commit 5db4d80c111c766076df8f5390452b61c84ab77f
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Wed May 4 19:48:56 2016 -0700

        IMPALA-3491: Use unique_database fixture in test_col_stats.py.

        The patch also addresses a TODO asking for
        test_col_stats.py to be merged into test_compute_stats.py

        Testing: I ran the test by itself in a loop 10 times,
        and the whole test_compute_stats.py locally. Also did
        a private core/hdfs run.

        Change-Id: I88aa77464a95993c018e19a52eeb496d7c3eef08
        Reviewed-on: http://gerrit.cloudera.org:8080/2963
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        alex.behm Alexander Behm added a comment - commit 5db4d80c111c766076df8f5390452b61c84ab77f Author: Alex Behm <alex.behm@cloudera.com> Date: Wed May 4 19:48:56 2016 -0700 IMPALA-3491 : Use unique_database fixture in test_col_stats.py. The patch also addresses a TODO asking for test_col_stats.py to be merged into test_compute_stats.py Testing: I ran the test by itself in a loop 10 times, and the whole test_compute_stats.py locally. Also did a private core/hdfs run. Change-Id: I88aa77464a95993c018e19a52eeb496d7c3eef08 Reviewed-on: http://gerrit.cloudera.org:8080/2963 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Internal Jenkins
        Hide
        alex.behm Alexander Behm added a comment -

        I fixed most tests, in particular the slowest ones. Considering this resolved.

        Show
        alex.behm Alexander Behm added a comment - I fixed most tests, in particular the slowest ones. Considering this resolved.

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development