Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4434

Improve auto-parallelism for tez

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.15.0
    • tez
    • None
    • Reviewed

    Description

      Tez auto-parallelism currently has some limitation:
      1. ShuffledVertexManager only decrease parallelism not increase
      2. Pig currently exaggerate parallelism at frontend, ShuffledVertexManager might get initial parallelism way large than actual, that would be costly

      Instead of that, we can gradually adjust initial vertex parallelism at runtime once upstream vertexes finishes.

      Attachments

        1. PIG-4434-5.patch
          63 kB
          Daniel Dai
        2. PIG-4434-4.patch
          61 kB
          Daniel Dai
        3. PIG-4434-3.patch
          57 kB
          Daniel Dai
        4. PIG-4434-2.patch
          53 kB
          Daniel Dai
        5. PIG-4434-1.patch
          46 kB
          Daniel Dai

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue TEZ-2234 Add API for statistics information - allow vertex managers to get output size per source vertex

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. TEZ-2232 Allow setParallelism to be called multiple times before tasks get scheduled

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. TEZ-2233 Allow EdgeProperty of an edge to be changed by VertexManager

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. TEZ-2242 Refactor ShuffleVertexManager code

          • Major - Major loss of function.
          • Closed
          is broken by

          Bug - A problem which impairs or prevents the functions of the product. TEZ-2310 Deadlock caused by StateChangeNotifier sending notifications on thread holding locks

          • Major - Major loss of function.
          • Closed

          Activity

            People

              daijy Daniel Dai
              daijy Daniel Dai
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: