[NEMO-352] Nemo on AWS Lambda

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: gsoc2019
• Assignee: Gao Zhiyuan
• Reporter: John Yang
• Created: 08/Mar/19 01:08
• Updated: 25/Aug/19 13:06

Description

AWS Lambda (and other cloud serverless frameworks) is increasingly widely used to run data analytics computations. So it'd be nice to enable Nemo to run on AWS Lambda, especially as one of the strengths of Nemo is the ease of enforcing resource-aware scheduling.

Although Nemo at the moment does support different types of resources such as transient and geo-distributed resources, these resources are assumed to be managed by a central resource manager and generally consumed by a Java process that starts with the standard main() without particular restriction.

However, these assumptions do not hold for the resources on AWS Lambda.

• Resources on AWS Lambda can only be used through deploying a function using a particular programming model and API enforced by AWS
• Functions running on AWS Lambda cannot communicate via network with each other directly
• Functions generally cannot share memory (e.g., shared variable)

Hence, a simple optimization pass such as TransientResourcePass or GeoDistPass would not suffice for Nemo to harness resources on AWS Lambda.

The following is a list of components/features that needs to be introduced to Nemo. (the list can grow of course as we develop and discover new things)

• Mechanisms to start/stop AWS Lambda
• AWS credentials management
• A new Nemo 'Executor' for AWS Lambda to run Nemo 'Task's
• Data (re)routing mechanisms for transferring data back and forth with AWS Lambda
• Scheduling mechanisms to (perhaps dynamically) decide which subset of computations to run on AWS Lambda

Issue Links

• is a parent of: NEMO-407 LambdaHandler for single-stage execution
• is a parent of: NEMO-385 Support Lambda Pass with lambda policy and lambda resource property
• is a parent of: NEMO-387 Support Lambda scheduler
• is a parent of: NEMO-398 ExecutorRepresenter interface and LambdaExecutorRepresenter
• is a parent of: NEMO-416 Guava vendor version conflict when deserializing Task object
• is a parent of: NEMO-417 User documentation on how to use LambdaExecutor

Sub-Tasks

• Support Lambda Pass with lambda policy and lambda resource property: Resolved
• Support Lambda scheduler: Resolved
• ExecutorRepresenter interface and LambdaExecutorRepresenter: Resolved
• LambdaHandler for single-stage execution: Open
• Multiple LambdaExecutor dispatching: Open