Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23831

Make auto reducer parallelism min threshold configurable.

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Hive on Tez disables auto-reduce and uses the maximum number of partitions when the minimum number of reducers, `estimated number * min.partition.factor`, is less than or equal to 1.

      https://github.com/apache/hive/blob/f98e136bdd5642e3de10d2fd1a4c14d1d6762113/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L121-L131

      https://issues.apache.org/jira/browse/HIVE-14200

       

      This can make Tez lose chance to optimize # of reducers especially in an environment with small `hive.tez.min.partition.factor` or big `hive.tez.max.partition.factor`.

      Attachments

        Issue Links

          Activity

            People

              okumin Shohei Okumiya
              okumin Shohei Okumiya
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: