Details
-
Umbrella
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
Description
A collection of ideas/issues for improving the data plane code.
High-level ideas:
- Explicit, deterministic threading (i.e., Code X is always executed by Thread Y): For example, in the control plane we have (1) application runner thread, (2) runtime master thread, and (3) scheduler runner thread. It is clear which part of code is executed by which thread.
- Clearer exception handling: Make it easy to see that an IO exception is guaranteed to propagate to a DataFetcher(= the Task Executor thread), and trigger task INPUT_READ_FAILURE, or TASK_OUTPUT_FAILURE. Provide enough information so the master knows which data blocks to recover.
- Provide simple interfaces to DataFetcher/TaskExecutor: A single, plain iterator would be the best (and not a Future of multiple iterators)
Attachments
Issue Links
- blocks
-
NEMO-137 Retry parent task(s) upon task INPUT_READ_FAILURE
- Open
1.
|
Simpler InputReader#read Return Type | Open | Unassigned | |
2.
|
Network threads should not touch BlockManagerMaster | Open | Unassigned | |
3.
|
Guava cache for BlockManagerWorker.pendingBlockLocationRequest | Open | Unassigned | |
4.
|
Decouple location request from actual data transfer | Open | Unassigned | |
5.
|
Integrate BlockTransferThrottler and backgroundExecutorService | Open | Unassigned | |
6.
|
Exception handling unit tests for the bytetransfer package | Open | Unassigned |