XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Tests
    • Labels:
      None
    • Target Version/s:

      Description

      When analyzing the test runtimes, there are many CliDriver tests that shows up as stragglers and are slow. Most of these tests are not really testing the execution engine. For example special_character_in_tabnames_1.q is the slowest test case that takes 419s in CliDriver but only 62s in MiniLlap. Similarly there are many test cases that can benefit from fast runtimes. We should consider moving the tests that are not testing the execution engine to MiniLlap (assuming it provides significant performance benefit).

      Here is the list of top 100 slow tests based on build #1055

      QFiles TestCliDriver elapsed time
      special_character_in_tabnames_1.q 419.229
      unionDistinct_1.q 278.583
      vector_leftsemi_mapjoin.q 232.313
      join_filters.q 172.436
      escape2.q 167.503
      archive_excludeHadoop20.q 163.522
      escape1.q 130.217
      lineage3.q 110.935
      insert_into_with_schema.q 107.345
      auto_join_filters.q 104.331
      windowing.q 99.622
      index_compact_binary_search.q 97.637
      cbo_rp_windowing_2.q 95.108
      vectorized_ptf.q 93.397
      dynpart_sort_optimization_acid.q 91.831
      partition_multilevels.q 90.392
      ptf.q 89.115
      sample_islocalmode_hook.q 88.293
      udaf_collect_set_2.q 84.725
      skewjoin.q 84.588
      lineage2.q 84.187
      correlationoptimizer1.q 80.367
      dynpart_sort_optimization.q 77.07
      orc_ppd_decimal.q 75.523
      orc_ppd_schema_evol_3a.q 75.352
      groupby_sort_skew_1_23.q 75.342
      cbo_rp_lineage2.q 75.283
      parquet_ppd_decimal.q 74.063
      sample_islocalmode_hook_use_metadata.q 73.988
      orc_analyze.q 73.803
      join_nulls.q 72.417
      semijoin.q 70.403
      correlationoptimizer6.q 69.151
      table_access_keys_stats.q 68.699
      autoColumnStats_2.q 68.632
      cbo_join.q 68.325
      cbo_rp_join.q 68.317
      sample10.q 64.513
      mergejoin.q 63.647
      multi_insert_move_tasks_share_dependencies.q 62.079
      union_view.q 61.772
      autoColumnStats_1.q 61.246
      groupby_sort_1_23.q 61.129
      pcr.q 59.546
      vectorization_short_regress.q 58.775
      auto_sortmerge_join_9.q 58.3
      correlationoptimizer2.q 56.591
      alter_merge_stats_orc.q 55.202
      vector_join30.q 54.85
      selectDistinctStar.q 53.981
      vector_decimal_udf.q 53.879
      auto_join30.q 53.762
      subquery_notin.q 52.879
      cbo_rp_subq_not_in.q 52.609
      cbo_rp_gby.q 51.866
      cbo_subq_not_in.q 51.672
      cbo_gby.q 50.361
      infer_bucket_sort.q 49.158
      ptf_streaming.q 48.484
      join_1to1.q 48.268
      load_dyn_part5.q 47.796
      limit_join_transpose.q 47.517
      ppd_windowing2.q 47.318
      dynpart_sort_opt_vectorization.q 47.208
      vector_number_compare_projection.q 47.024
      correlationoptimizer4.q 45.472
      orc_ppd_date.q 45.19
      global_limit.q 44.438
      union_top_level.q 44.229
      llap_partitioned.q 44.139
      orc_ppd_timestamp.q 43.617
      parquet_ppd_date.q 43.539
      multiMapJoin2.q 43.036
      parquet_ppd_timestamp.q 42.665
      vector_partitioned_date_time.q 42.511
      auto_sortmerge_join_8.q 42.377
      create_view.q 42.23
      windowing_windowspec2.q 42.202
      multiMapJoin1.q 41.176
      vector_decimal_2.q 41.026
      bucket_groupby.q 40.565
      rcfile_merge2.q 39.782
      index_compact_2.q 39.765
      join_nullsafe.q 39.698
      vector_join_filters.q 39.343
      cbo_rp_auto_join1.q 39.308
      vector_auto_smb_mapjoin_14.q 39.17
      vector_udf1.q 38.988
      rcfile_createas1.q 38.932
      cbo_rp_semijoin.q 38.675
      auto_join_nulls.q 38.519
      cbo_rp_unionDistinct_2.q 37.815
      union_remove_26.q 37.672
      rcfile_merge3.q 37.373
      rcfile_merge4.q 37.194
      bucketsortoptimize_insert_2.q 37.187
      cbo_limit.q 37.038
      auto_sortmerge_join_6.q 36.663
      join43.q 36.656

        Attachments

        1. HIVE-14877.1.patch
          3.69 MB
          Prasanth Jayachandran
        2. HIVE-14877.2.patch
          3.68 MB
          Prasanth Jayachandran
        3. HIVE-14877.3.patch
          3.68 MB
          Prasanth Jayachandran
        4. HIVE-14877.4.patch
          3.82 MB
          Prasanth Jayachandran
        5. HIVE-14877.5.patch
          3.82 MB
          Prasanth Jayachandran
        6. HIVE-14877.5.patch
          3.82 MB
          Prasanth Jayachandran
        7. HIVE-14877.6.patch
          3.82 MB
          Prasanth Jayachandran

          Issue Links

            Activity

              People

              • Assignee:
                prasanth_j Prasanth Jayachandran
                Reporter:
                prasanth_j Prasanth Jayachandran
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: