[WHIRR-506] Support elastic mapreduce in whirr

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: cli, core
• Labels: gsoc gsoc2012 mentor
• Assignee: Unassigned
• Reporter: David Alves
• Created: 07/Feb/12 22:33
• Updated: 27/Mar/12 04:57

Description

This is to be an umbrella issue in the path towards having multicloud EMR with whirr.

Some of the things that must happen towards multicloud EMR (as discussed in IRC):

• Hadoop deployment must be "rock solid"
• Submitting and monitoring an hadoop mapreduce job through whirr
• distcp from blobstore to hadoop/hbase cluster
• cli component for job submission and monitoring.

Some of the things that would be nice to have additionally:

• pig service
• hive service
• sqoop service
• regular+spot instances in EMR
• multistage provisioning (different cluster sizes for different phases)

Issue Links

• depends upon: WHIRR-214 Add/Remove nodes to/from running clusters
• depends upon: WHIRR-567 Implement distcp in whirr
• is related to: WHIRR-529 GSoC 2012: Add new services to Apache Whirr