Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22942

Replace PTest with an alternative

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      I never opened a jira about this...but it might actually help collect ideas and actually start going somewhere sooner than later

      Right now we maintain the ptest2 project inside Hive to be able to run Hive tests in a distributed fashion...the backstab of this solution is that we are putting much effort into maintaining a distributed test execution framework...

      I think it would be better if we could find an off the shelf solution for the task and migrate to that instead of putting more efforts into the ptest framework

      some info/etc about how it compares to existing one:
      https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit#heading=h.e51vlxui3e6n

      Attachments

        1. HIVE-22942.01.patch
          25 kB
          Zoltan Haindrich

        Issue Links

        1.
        CliDriver leaves the session id in the threadname on failure Sub-task Closed Zoltan Haindrich   Actions
        2.
        UDAFExampleMaxMinNUtil.sortedMerge must be able to handle all inputs Sub-task Closed Zoltan Haindrich   Actions
        3.
        Prevent infinite loop in TestJdbcWithMiniLlapArrow Sub-task Closed Zoltan Haindrich   Actions
        4.
        Reduce MiniDruidCluster memory requeirements Sub-task Closed Zoltan Haindrich   Actions
        5.
        PostExecOrcFileDump listing order may depend on the underlying filesystem Sub-task Closed Zoltan Haindrich   Actions
        6.
        Increase timeout for some tez tests Sub-task Closed Zoltan Haindrich   Actions
        7.
        avro-mapred should not pull in org.mortbay.jetty Sub-task Closed Zoltan Haindrich   Actions
        8.
        Scheduled query related qtests may not finish before it's expected Sub-task Closed Zoltan Haindrich   Actions
        9.
        Provide a way to have only a selection of datasets loaded Sub-task Closed Zoltan Haindrich   Actions
        10.
        Change spark related tests to be optional Sub-task Closed Zoltan Haindrich   Actions
        11.
        Update some tests to better tolerate a more busy environment Sub-task Closed Zoltan Haindrich   Actions
        12.
        Tolerate in tests that metastore is not removing rows from the skewed_string_list_values table Sub-task Closed Zoltan Haindrich   Actions
        13.
        Add qsplits profile Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        Actions
        14.
        schq_ingest may run twice during a test execution Sub-task Closed Zoltan Haindrich   Actions
        15.
        QueryDisplay must be threadsafe Sub-task Closed Zoltan Haindrich   Actions
        16.
        Add qoption to disable qtests Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        Actions
        17.
        Many fixes and improvements to stabilize tests Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        18.
        Upgrade to Kudu 1.12 Sub-task Closed Zoltan Haindrich   Actions
        19.
        Close stale PRs automatically Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        Actions
        20.
        Build failure is incorrectly reported as tests passed Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        Actions
        21.
        Use junit5 to execute tests Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        22.
        Enforce ASF headers on source files Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        Actions
        23.
        Use the test target instead of install Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        24.
        Provide a way to check test stability Sub-task Closed Zoltan Haindrich   Actions
        25.
        TestTxnHandler#allocateNextWriteIdRetriesAfterDetectingConflictingConcurrentInsert is unstable Sub-task Resolved Marton Bod

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        26.
        Early abort the build in case new commits are added to the PR Sub-task Resolved Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        27.
        RetryTest is unstable Sub-task Closed Aasha Medhi

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        28.
        Move shaded perf builds into a separate profile Sub-task Resolved Zoltan Haindrich   Actions
        29.
        TestAcidTxnCleanerService is unstable Sub-task Closed Peter Varga

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 50m
        Actions
        30.
        special_character_in_tabnames_1.q is unstable Sub-task Resolved Krisztian Kasa   Actions
        31.
        Run metastore verification tests during precommit Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h
        Actions
        32.
        Some jdbc tests are unstable Sub-task Resolved Jesús Camacho Rodríguez   Actions
        33.
        Enable removal of old builds on hive ci Sub-task Closed Zoltan Haindrich

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        34.
        TestBeeLineWithArgs#testRowsAffected is unstable Sub-task Closed Unassigned   Actions
        35.
        TestScheduledReplicationScenarios#testAcidTablesReplLoadBootstrapIncr is unstable Sub-task Open Pravin Sinha

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        36.
        Provide a way to include/exclude a set of tests Sub-task Open Zoltan Haindrich   Actions
        37.
        Collect ignored tests Sub-task Open Unassigned   Actions
        38.
        Run metastore integration tests during precommit Sub-task Open Unassigned   Actions
        39.
        Running masking_mv.q breaks strict_pruning.q Sub-task Patch Available Zoltan Haindrich   Actions
        40.
        stats_list_bucketing breaks sysdb output Sub-task Open Zoltan Haindrich   Actions
        41.
        Rows are not removed from the skewed_string_list_values table Sub-task Open Unassigned   Actions
        42.
        TestJdbcGenericUDTFGetSplits2#testGenericUDTFOrderBySplitCount1 is flaky Sub-task Resolved Simhadri Govindappa

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 4h 40m
        Actions
        43.
        TestSessionManagerMetrics#testAbandonedSessionMetrics is unstable Sub-task Open Unassigned   Actions
        44.
        TransactionalKafkaWriterTest#writerFencedOut is unstable Sub-task Open Unassigned   Actions
        45.
        TestDbNotificationListener#sqlInsertPartition is unstable Sub-task Open Ashish Sharma   Actions
        46.
        strict_pruning_2 is unstable Sub-task Open Unassigned   Actions
        47.
        TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is flaky Sub-task Open Unassigned   Actions
        48.
        TestTriggersTezSessionPoolManager sometimes exits the JVM Sub-task Open Unassigned   Actions
        49.
        TestNewGetSplitsFormat is unstable Sub-task Open Unassigned   Actions
        50.
        TestJdbcWithServiceDiscovery is unstable Sub-task Open Unassigned   Actions
        51.
        TestActivePassiveHA is unstable Sub-task Open KIRTI RUGE

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 20m
        Actions
        52.
        TestJdbcWithMiniLlapVectorArrow is unstable Sub-task Open Unassigned   Actions
        53.
        TestMiniLlapCliDriver.testCliDriver[merge_test_dummy_operator] is unstable Sub-task Open Unassigned   Actions
        54.
        TestSlotZnode concurrency tests are unstable Sub-task Open Unassigned   Actions
        55.
        temp_table_multi_insert_partitioned.q is unstable Sub-task Open Unassigned   Actions
        56.
        schema_evol_par_vec_table_* tests are unstable Sub-task Open Unassigned   Actions
        57.
        authorization_disallow_transform.q is unstable Sub-task Open Unassigned

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        Actions
        58.
        Fix orc_merge9 Sub-task Resolved Krisztian Kasa

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        Actions
        59.
        Run a smoketest during precommit Sub-task Open Unassigned   Actions
        60.
        TestSparkClient is flaky Sub-task Open Unassigned   Actions
        61.
        TestDbNotificationListener is unstable Sub-task Reopened KIRTI RUGE   Actions
        62.
        TestTriggersMoveWorkloadManager is unstable Sub-task Open Unassigned   Actions
        63.
        TestMetrics is flaky Sub-task Open Unassigned

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        64.
        TestNegativeCliDriver#[external_jdbc_negative] is flaky Sub-task Open KIRTI RUGE   Actions
        65.
        TestMiniLlapLocalCliDriver#testCliDriver[schq_materialized] is flaky Sub-task Open KIRTI RUGE

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        Actions
        66.
        TestCodahaleMetrics.testFileReporting is flaky Sub-task Open Unassigned   Actions
        67.
        HiveKafkaProducerTest.resumeTransaction is flaky Sub-task Closed Akshat Mathur   Actions
        68.
        TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic] resultset unstable Sub-task Open Unassigned   Actions
        69.
        jdbc_handler is flaky Sub-task Open KIRTI RUGE   Actions
        70.
        Run Oracle integration tests during precommit Sub-task Open Unassigned   Actions
        71.
        Run Mssql integration tests during precommit Sub-task Open Unassigned   Actions
        72.
        Remove unused code from ptest/ptest2 Sub-task Closed Stamatis Zampetakis

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kgyrtkirk Zoltan Haindrich Assign to me
            kgyrtkirk Zoltan Haindrich
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 23h 20m
              23h 20m

              Slack

                Issue deployment