Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-234

Implement TransformChaining

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      In nemo, an upstream Transform emits data to the data queue of the downstream Transform, and the downstream Transform retrieves the data from the data queue, even though they are in the same stage.

       
      ex)

      Transform1 ==> emit data to a list ==> Transform2 deques the data from the list.

       

      This requires enqueue/dequeue overhead but we can optimize this by directly sending data to the next Transform, by chaining the Transforms in the same stage.

      ex)

      Transform1 ==> directly call Transform2.onData()

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                taegeonum Tae-Geon Um
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: