Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-410

Refactor EdgeProperties to be more clear

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2.0
    • Component/s: None
    • Labels:
      None

      Description

      During discussion with users there was feedback that edge properties need to be named better to make them more clear. There was a suggestion to look at MPI for inspiration. Based on that feedback, the proposal is to renamed ConnectionPattern to DataMovement as that is essentially what the property is defining. A Bipartite connection pattern can be constructed from both broadcast and scatter-gather data movement types. There will be 3 kinds of data movements initially.
      ONE_TO_ONE - Defines an output produced by the ith upstream task is available the the ith downstream task.
      BROADCAST - Defines an output produced by any upstream task is available to all downstream tasks.
      SCATTER_GATHER - Defines that the ith output produced by all upstream tasks is available to the same downstream task. Upstream tasks scatter there outputs and they are gathered by designated downstream tasks.
      To be clear, output being available to the a task does not imply that the entire output is transferred/read by it. The task can choose to read any amount of the total data.

      Current users: In the EdgeProperty object
      Please change EdgeConnectionPattern.BIPARTITE -> DataMovementType.SCATTER_GATHER
      Please change SourceType.STABLE -> DataSourceType.PERSISTED
      Please add SchedulingType.SEQUENTIAL to EdgeProperty objects.
      The getter methods have similar name changes.

        Attachments

        1. TEZ-410.1.patch
          26 kB
          Bikas Saha
        2. TEZ-410.2.patch
          37 kB
          Bikas Saha
        3. TEZ-410.3.patch
          44 kB
          Bikas Saha
        4. TEZ-410.4.patch
          49 kB
          Bikas Saha
        5. TEZ-410.5.patch
          54 kB
          Bikas Saha

          Issue Links

            Activity

              People

              • Assignee:
                bikassaha Bikas Saha
                Reporter:
                bikassaha Bikas Saha
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: