Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-24485

Make the slow-start behavior tunable

Log workAgile BoardRank to TopRank to BottomAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.2, 4.0.0
    • Fix Version/s: None
    • Component/s: Hive, Tez

      Description

      This ticket would enable users to configure the timing of slow-start with `tez.shuffle-vertex-manager.min-src-fraction` and `tez.shuffle-vertex-manager.max-src-fraction`.
      Hive on Tez currently doesn't honor these parameters and ShuffleVertexManager always uses the default value.
      We can control the timing to start vertexes the accuracy of estimated input size if we can tweak these ones. This is useful when a vertex has tasks that process a different amount of data.

       

      We can reproduce the issue with this query.

      SET hive.tez.auto.reducer.parallelism=true;
      SET hive.tez.min.partition.factor=1.0; -- enforce auto-parallelism
      SET tez.shuffle-vertex-manager.min-src-fraction=0.55;
      SET tez.shuffle-vertex-manager.max-src-fraction=0.95;
      CREATE TABLE mofu (name string);
      INSERT INTO mofu (name) VALUES ('12345');
      SELECT name, count(*) FROM mofu GROUP BY name;

      The fractions are ignored.

      2020-12-04 11:41:42,484 [INFO] [Dispatcher thread {Central}] |vertexmanager.ShuffleVertexManagerBase|: Settings minFrac: 0.25 maxFrac: 0.75 auto: true desiredTaskIput: 256000000
      

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 40m
              40m

                Issue deployment