Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1152 Optimize broadcast join for scalability
  3. TEZ-1157

Optimize broadcast :- Tasks pertaining to same job in same machine should not download multiple copies of broadcast data

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.1
    • None

    Description

      Currently tasks (belonging to same job) running in the same machine download its own copy of broadcast data. Optimization could be to download one copy in the machine, and the rest of the tasks can refer to this downloaded copy.

      (results after this feature)

      Attachments

        1. TEZ-1152.WIP.patch
          17 kB
          Rajesh Balamohan
        2. TEZ-broadcast-shuffle+vertex-parallelism.patch
          14 kB
          Gopal Vijayaraghavan
        3. TEZ-1157.3.WIP.patch
          33 kB
          Gopal Vijayaraghavan
        4. TEZ-1157.4.WIP.patch
          36 kB
          Gopal Vijayaraghavan
        5. TEZ-1157.5.WIP.patch
          35 kB
          Gopal Vijayaraghavan
        6. TEZ-1157.6.patch
          37 kB
          Gopal Vijayaraghavan
        7. TEZ-1157.7.patch
          35 kB
          Gopal Vijayaraghavan
        8. TEZ-1157.8.patch
          35 kB
          Gopal Vijayaraghavan
        9. TEZ-1157.9.patch
          35 kB
          Gopal Vijayaraghavan
        10. latency.png
          67 kB
          Gopal Vijayaraghavan
        11. connections.png
          77 kB
          Gopal Vijayaraghavan
        12. TEZ-1157.10.patch
          36 kB
          Gopal Vijayaraghavan

        Issue Links

        is blocked by

        Improvement - An improvement or enhancement to an existing feature or task. TEZ-1537 Expose edge information in I/P/IContexts

        • Major - Major loss of function.
        • Open
        relates to

        Bug - A problem which impairs or prevents the functions of the product. TEZ-1360 Provide vertex parallelism to each vertex task

        • Major - Major loss of function.
        • Closed

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gopalv Gopal Vijayaraghavan
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment