Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4130

Config for hard limiting the number of splits

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      During the investigation of a customer issue, I found that tez generated a dag plan containing >4k tasks. It failed for hive because of bucket number limitations (4k). It can be configured properly, e.g. bigger splits (tez.grouping.min-size), but maybe it would be more convenient for users to config a hard limit for the number of splits.

      Attachments

        1. TEZ-4130.01.patch
          1 kB
          László Bodor
        2. TEZ-4130.02.patch
          1 kB
          László Bodor

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. TEZ-4271 Add config to limit desiredNumSplits

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. TEZ-4271 Add config to limit desiredNumSplits

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              abstractdog László Bodor
              abstractdog László Bodor
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: