Create service interface for job submission

• Key: MAPREDUCE-454
• Type: New Feature
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Jeff Hammerbacher

Description

Porting a discussion from the LinkedIn Hadoop group to the Hadoop JIRA: http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&gid=988957&discussionID=2156671&sik=1239077959330

Comments

[Jeff Hammerbacher]

A comment from Brian Starke on the LinkedIn group:

There's nothing that we found, so we ended up making our own service layer that implements ServiceTool and Configurable for job creation and submission.

It was a little bit of a pain in the ass, but now we have a full on service layer on top of our Hadoop clusters, complete with queuing, email alerting, scheduling, and even a dashboard. Well worth the effort, although I'm a little surprised no one has a project like this open sourced yet. Ours isn't generic enough to open source just yet, but now that I know at least one other person might find this useful - I'll spend some cycles seeing if we can get around to getting this out there.

[Steve Loughran]

+1 to a remote job submission interface. My requirements

1. RESTful, long-haul, stable interface
2. Some form of notification on job completion. This could be polling the Atom feed of state changes.

HADOOP-4559 relates to this

[Joydeep Sen Sarma]

+1. I would love to see something EMR style api that abstracts away some of the exact details of the cluster being used - specifically the mapred jobtrcaker being used. the idea being that if there are multiple mapred clusters - we can have some control over what jobs go where.

i don't think this is a dup of 5821. would love to help out if there was a patch available ..