Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-2962

Use per partition stats in shuffle vertex manager auto parallelism

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      The original code used output size sent by completed tasks. Recently per partition stats have been added that provide granular information. Using partition stats may be more accurate and also remove the duplicate counting of data size in partition stats and per task overall.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. TEZ-3126 Log reason for not reducing parallelism

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. TEZ-3303 Have ShuffleVertexManager consume more precise partition stats

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              bikassaha Bikas Saha
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: