Web service interface to the JobTracker

• Key: MAPREDUCE-445
• Type: New Feature
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Owen O'Malley

Description

I think we need to provide a web services interface to submit and track jobs. This will simplify cross-version and non-Java access to JobTracker functionality.

Comments

[Steve Loughran]

+1 to a RESTy long-haul API with status served as Atom feeds.

[Steve Loughran]

Thought it was familiar

[Jeff Hammerbacher]

Hey Owen, this is a dupe of https://issues.apache.org/jira/browse/HADOOP-5633

[Joydeep Sen Sarma]

i am a little surprised this is considered dup of 5633. what i would love to see is a web service api to submit jobs that is jobtracker agnostic. that way we can control what jobs go where (in the (not so uncommon) case of multiple map-red clusters attached to the same hdfs). (so almost EMR style)

[Steve Loughran]

It's all the same need

For me

• Works long-haul, which either means slow networks or complex networks
• Server can be NAT-ed to the front, no require for 1:1 match of IP addresses on both sides
• Not brittle against Hadoop versions (i.e. stable and version checking (in the URL?)
• uses HTTPs with various authentication opens (certificates as well as passwords)
• has a java client
• has a client in something like python just to test x-platform nature
• has an ant task that uses that java client so you can easily submit work
• Usable by humans too (gui, list jobs, form submit, &c)

I would like to get involved in this, once I've got other patches checked in. It is a need of mine.