Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-753 [Umbrella] Scalability improvements
  3. TEZ-646

Avoid creating multiple copies of the same Event payload

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 0.3.0
    • None
    • None

    Description

      OnFileSortedOutput generates the same event payload for all downstream tasks. As an example, for a simple MR job - the number of copies of this is equal to the number of reduce tasks.

      This needs to be done in a clean manner though - since the event model is meant to generate a separate payload for each downstream task.

      Attachments

        1. TEZ-646.1.txt
          19 kB
          Siddharth Seth
        2. TEZ-646.2.txt
          27 kB
          Siddharth Seth
        3. TEZ-646.3.txt
          26 kB
          Siddharth Seth

        Activity

          People

            sseth Siddharth Seth
            sseth Siddharth Seth
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: