[SPARK-10171] AWS Lambda Executors

• Type: Wish
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Jaka Jancar
• Created: 22/Aug/15 20:35
• Updated: 02/Sep/15 12:17
• Resolved: 02/Sep/15 12:17

Description

It would be great if Spark supported using AWS Lambda for execution in addition to Standalone, Mesos and YARN, getting rid of the concept of a "cluster" and having a single infinite-sized one.

Couple of problems I see today:

• Execution time is limited to 60s. This will probably change in the future.
• "Burstiness" is still not very high.