[NEMO-449] Runtime Level Caching Mechanism

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: gsoc2023
• Assignee: Unassigned
• Reporter: Dongjoo Lee
• Created: 17/Jun/20 07:18
• Updated: 09/Mar/23 08:53

Description

If the the compile time identifies what data can be cached, the runtime requires logic to make this happen.

Implementation needs:

• (Driver) receive and update the status of blocks from various Executors, right now this seems to be best implemented as part of BlockManagerMaster
• (Driver) communicate to the  Executors the availability, location and status of blocks
• Possible concurrency issues:

1. Concurrency in Driver when multiple Executors update/inquire the same block information
2. Concurrency in Executor when a single cached block is accessed simultaneously.

 

Issue Links

• incorporates: NEMO-450 Block status aware Master
• incorporates: NEMO-451 SizeEstimator logic for determining size of java objects produced at runtime
• is part of: NEMO-444 Implement spill mechanism on Nemo