[WHIRR-119] Job Submission and dynamic provisioning framework for Hadoop Clouds

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.2.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Krishna Sankar
• Created: 28/Oct/10 02:34
• Updated: 29/Oct/10 18:37

Description

A thin framework that can submit a MR job, run it and report results. Some thoughts:

1. Most probably it will be a server-side daemon
2. JSON over HTTP with REST semantics
3. Functions - top level preliminary
   1. Accept a job and it's components at a well known URL
   2. Parse & create MR workflow
   3. Create & store a job context - ID, security artifacts et al
   4. Return a status URL (can be used to query status or kill the job) This is the REST model
   5. Run the job (might include dynamic elastic cloud provisioning for example OpenStack)
   6. As the job runs, collect and store in the job context
   7. If client queries return status
   8. Once job is done, store status and return results (most probably pointers to files and so forth)
   9. Calculate & store performance metrics
   10. Calculate & store charge back in generic units (eg: CPU,Memory,Network,storage
   11. As and when the client asks, return job results
4. Some thoughts on implementation
   1. Store context et al in HBase
   2. A Clojure implementation ?
   3. Packaging like OVF ? (with embedded pointers to VM, data and so forth)
   4. For 1st release assume a homogeneous Hadoop infrastructure in a cloud
   5. Customer reporter/context counters?
   6. Distributed cache for framework artifacts and run time monitoring ?
   7. Most probably might have to use taskrunner ?
   8. Extend classes with submission framework setup and teardown code ?