[NEMO-144] Improve Data Plane Code

• Type: Umbrella
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Labels: runtime
• Assignee: Unassigned
• Reporter: John Yang
• Created: 08/Jul/18 06:09
• Updated: 09/Aug/18 05:11

Description

A collection of ideas/issues for improving the data plane code. 

High-level ideas:

• Explicit, deterministic threading (i.e., Code X is always executed by Thread Y): For example, in the control plane we have (1) application runner thread, (2) runtime master thread, and (3) scheduler runner thread. It is clear which part of code is executed by which thread.
• Clearer exception handling: Make it easy to see that an IO exception is guaranteed to propagate to a DataFetcher(= the Task Executor thread), and trigger task INPUT_READ_FAILURE, or TASK_OUTPUT_FAILURE. Provide enough information so the master knows which data blocks to recover.
• Provide simple interfaces to DataFetcher/TaskExecutor: A single, plain iterator would be the best (and not a Future of multiple iterators)

 

Issue Links

• blocks: NEMO-137 Retry parent task(s) upon task INPUT_READ_FAILURE
• contains: NEMO-177 Fix BytesDecoder to handle empty stream properly
• contains: NEMO-176 Improve sequential read from disk
• contains: NEMO-51 Intermediate data location aware scheduling
• contains: NEMO-172 Implement one partition per one element partitioner

Sub-Tasks

• Simpler InputReader#read Return Type: Open
• Network threads should not touch BlockManagerMaster: Open
• Guava cache for BlockManagerWorker.pendingBlockLocationRequest: Open
• Decouple location request from actual data transfer: Open
• Integrate BlockTransferThrottler and backgroundExecutorService: Open
• Exception handling unit tests for the bytetransfer package: Open