Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3286

Allow clients to set processor reserved memory per vertex (instead of per container)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.3
    • Fix Version/s: 0.9.0, 0.8.4
    • Component/s: None
    • Labels:
      None

      Description

      tez.task.scale.memory.reserve-fraction can be set by clients to control how much memory is available to the processor. Ths values applies at a container level though, instead of at a vertex level.
      In case of a hash-join - the processor typically needs more memory. In case of a Shuffle join - the processor may not need as much. In DAGs with a mix of map joins and shuffle joins - setting this at a container level is sub-optimal.

      To a large extent this comes down to propagating vertex configs to the container / task.

        Attachments

        1. TEZ-3286.3.withTestTimeout.txt
          40 kB
          Siddharth Seth
        2. TEZ-3286.3.patch
          40 kB
          Hitesh Shah
        3. TEZ-3286.3.branch-07.patch
          34 kB
          Hitesh Shah
        4. TEZ-3286.2.patch
          37 kB
          Hitesh Shah
        5. TEZ-3286.1.patch
          30 kB
          Hitesh Shah

          Issue Links

            Activity

              People

              • Assignee:
                hitesh Hitesh Shah
                Reporter:
                wzheng Wei Zheng
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: