Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-144

Improve Data Plane Code

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      A collection of ideas/issues for improving the data plane code. 

      High-level ideas:

      • Explicit, deterministic threading (i.e., Code X is always executed by Thread Y): For example, in the control plane we have (1) application runner thread, (2) runtime master thread, and (3) scheduler runner thread. It is clear which part of code is executed by which thread.
      • Clearer exception handling: Make it easy to see that an IO exception is guaranteed to propagate to a DataFetcher(= the Task Executor thread), and trigger task INPUT_READ_FAILURE, or TASK_OUTPUT_FAILURE. Provide enough information so the master knows which data blocks to recover.
      • Provide simple interfaces to DataFetcher/TaskExecutor: A single, plain iterator would be the best (and not a Future of multiple iterators)

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                johnyangk John Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: