Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22942

Replace PTest with an alternative

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: None

      Description

      I never opened a jira about this...but it might actually help collect ideas and actually start going somewhere sooner than later

      Right now we maintain the ptest2 project inside Hive to be able to run Hive tests in a distributed fashion...the backstab of this solution is that we are putting much effort into maintaining a distributed test execution framework...

      I think it would be better if we could find an off the shelf solution for the task and migrate to that instead of putting more efforts into the ptest framework

      some info/etc about how it compares to existing one:
      https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit#heading=h.e51vlxui3e6n

        Attachments

        1. HIVE-22942.01.patch
          25 kB
          Zoltan Haindrich

          Issue Links

          1.
          CliDriver leaves the session id in the threadname on failure Sub-task Resolved Zoltan Haindrich  
          2.
          UDAFExampleMaxMinNUtil.sortedMerge must be able to handle all inputs Sub-task Resolved Zoltan Haindrich  
          3.
          Prevent infinite loop in TestJdbcWithMiniLlapArrow Sub-task Resolved Zoltan Haindrich  
          4.
          Reduce MiniDruidCluster memory requeirements Sub-task Resolved Zoltan Haindrich  
          5.
          PostExecOrcFileDump listing order may depend on the underlying filesystem Sub-task Resolved Zoltan Haindrich  
          6.
          Increase timeout for some tez tests Sub-task Resolved Zoltan Haindrich  
          7.
          avro-mapred should not pull in org.mortbay.jetty Sub-task Resolved Zoltan Haindrich  
          8.
          Scheduled query related qtests may not finish before it's expected Sub-task Resolved Zoltan Haindrich  
          9.
          Provide a way to have only a selection of datasets loaded Sub-task Resolved Zoltan Haindrich  
          10.
          Change spark related tests to be optional Sub-task Resolved Zoltan Haindrich  
          11.
          Update some tests to better tolerate a more busy environment Sub-task Resolved Zoltan Haindrich  
          12.
          Tolerate in tests that metastore is not removing rows from the skewed_string_list_values table Sub-task Resolved Zoltan Haindrich  
          13.
          Add qsplits profile Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          14.
          schq_ingest may run twice during a test execution Sub-task Resolved Zoltan Haindrich  
          15.
          QueryDisplay must be threadsafe Sub-task Resolved Zoltan Haindrich  
          16.
          Add qoption to disable qtests Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          17.
          Many fixes and improvements to stabilize tests Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          18.
          Upgrade to Kudu 1.12 Sub-task Resolved Zoltan Haindrich  
          19.
          Close stale PRs automatically Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          20.
          Build failure is incorrectly reported as tests passed Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          21.
          Use junit5 to execute tests Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          22.
          Enforce ASF headers on source files Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          23.
          Use the test target instead of install Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          24.
          Provide a way to check test stability Sub-task Resolved Zoltan Haindrich  
          25.
          TestTxnHandler#allocateNextWriteIdRetriesAfterDetectingConflictingConcurrentInsert is unstable Sub-task Resolved Marton Bod

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          26.
          Early abort the build in case new commits are added to the PR Sub-task Resolved Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          27.
          RetryTest is unstable Sub-task Resolved Aasha Medhi

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          28.
          Move shaded perf builds into a separate profile Sub-task Resolved Zoltan Haindrich  
          29.
          Provide a way to include/exclude a set of tests Sub-task Open Zoltan Haindrich  
          30.
          Collect ignored tests Sub-task Open Unassigned  
          31.
          Run metastore integration tests during precommit Sub-task Open Unassigned  
          32.
          Running masking_mv.q breaks strict_pruning.q Sub-task Patch Available Zoltan Haindrich  
          33.
          stats_list_bucketing breaks sysdb output Sub-task Open Zoltan Haindrich  
          34.
          Rows are not removed from the skewed_string_list_values table Sub-task Open Unassigned  
          35.
          TestJdbcGenericUDTFGetSplits2#testGenericUDTFOrderBySplitCount1 is flaky Sub-task Open Unassigned  
          36.
          TestScheduledReplicationScenarios#testAcidTablesReplLoadBootstrapIncr is unstable Sub-task Open Pravin Sinha

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          37.
          TestBeeLineWithArgs#testRowsAffected is unstable Sub-task Open Unassigned  
          38.
          TestSessionManagerMetrics#testAbandonedSessionMetrics is unstable Sub-task Open Unassigned  
          39.
          TransactionalKafkaWriterTest#writerFencedOut is unstable Sub-task Open Unassigned  
          40.
          TestDbNotificationListener#sqlInsertPartition is unstable Sub-task Open Unassigned  
          41.
          strict_pruning_2 is unstable Sub-task Open Unassigned  
          42.
          TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is flaky Sub-task Open Jesus Camacho Rodriguez  
          43.
          TestTriggersTezSessionPoolManager sometimes exits the JVM Sub-task Open Unassigned  
          44.
          TestNewGetSplitsFormat is unstable Sub-task Open Unassigned  
          45.
          TestAcidTxnCleanerService is unstable Sub-task Resolved Peter Varga

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          46.
          TestJdbcWithServiceDiscovery is unstable Sub-task Open Unassigned  
          47.
          TestActivePassiveHA is unstable Sub-task Open Unassigned  
          48.
          TestJdbcWithMiniLlapVectorArrow is unstable Sub-task Open Unassigned  
          49.
          TestMiniLlapCliDriver.testCliDriver[merge_test_dummy_operator] is unstable Sub-task Open Unassigned  
          50.
          TestSlotZnode concurrency tests are unstable Sub-task Open Unassigned  
          51.
          temp_table_multi_insert_partitioned.q is unstable Sub-task Open Unassigned  
          52.
          schema_evol_par_vec_table_* tests are unstable Sub-task Open Unassigned  
          53.
          authorization_disallow_transform.q is unstable Sub-task Open Unassigned  
          54.
          special_character_in_tabnames_1.q is unstable Sub-task Resolved Krisztian Kasa  
          55.
          Fix orc_merge9 Sub-task Open Unassigned  
          56.
          Run a smoketest during precommit Sub-task Open Unassigned  
          57.
          TestSparkClient is flaky Sub-task Open Unassigned  
          58.
          TestDbNotificationListener is unstable Sub-task Open Aasha Medhi  
          59.
          TestTriggersMoveWorkloadManager is unstable Sub-task Open Unassigned  
          60.
          TestMetrics is flaky Sub-task Open Unassigned  
          61.
          TestNegativeCliDriver#[external_jdbc_negative] is flaky Sub-task Open Unassigned  
          62.
          TestMiniLlapLocalCliDriver#testCliDriver[schq_materialized] is flaky Sub-task Open Zoltan Haindrich  
          63.
          TestCodahaleMetrics.testFileReporting is flaky Sub-task Open Unassigned  
          64.
          HiveKafkaProducerTest.resumeTransaction is flaky Sub-task Open Unassigned  
          65.
          TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic] resultset unstable Sub-task Open Unassigned  
          66.
          jdbc_handler is flaky Sub-task Open Unassigned  
          67.
          Run metastore verification tests during precommit Sub-task Open Zoltan Haindrich

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          68.
          Some jdbc tests are unstable Sub-task Resolved Jesus Camacho Rodriguez  

            Activity

              People

              • Assignee:
                kgyrtkirk Zoltan Haindrich
                Reporter:
                kgyrtkirk Zoltan Haindrich
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 12h
                  12h