Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3206

Have unordered partitioned KV output send partition stats via VertexManagerEvent

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9.0
    • None
    • None

    Description

      As part of the auto-parallelism feature, ordered partitioned KV output's partition stats are sent to ShuffleVertexManager via VertexManagerEvent. But this isn't available for unordered partitioned output. Having UnorderedPartitionedKVWriter send partition stats will enable the auto-parallelism support for unordered KV or other custom data routing mechanisms that depend on partition size.

      Attachments

        1. TEZ-3206.patch
          21 kB
          Ming Ma
        2. TEZ-3206-2.patch
          30 kB
          Ming Ma
        3. TEZ-3206-3.patch
          30 kB
          Ming Ma
        4. TEZ-3206-4.patch
          30 kB
          Ming Ma

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. TEZ-3209 Support for fair custom data routing

          • Major - Major loss of function.
          • Open

          Activity

            People

              mingma Ming Ma
              mingma Ming Ma
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: