Harsh - I'm not familiar with the HBase case; can you please add more colour?
In this case, won't it be sufficient to schedule maps on the RS? If the data is already sorted, but would you try schedule reduces instead?
We have this concept of bulkloads, for example, in HBase, where the Maps read in data from a raw source (such as a delimited text file) and passes it to a reducer (partitioned by TotalOrderPartitioner based on the region distribution of the table in HBase). The sorted data is then written onto a file on HDFS and later, injected into the /hbase directory structure for serving.
There's cheap gains (but gains nevertheless) if the data written by the reducer is local to the RegionServer hosting that specific partition (region) itself, before we bulkload it in.
Likewise, if people have HBase jobs doing a reduce phase for whatever reason, and wish to achieve locality such that the reducer task (which emit the keys) are local to the regionserver serving the same region for those keys, they can do so via a pre-configured job.
There are some use-cases out of HBase as well (I'll let those who've desired this comment), but maybe YARN can change those to be outside of MR today.
Or maybe HBase can get a custom AM to do their work in more efficient manner than the current MR (MR is easy to use though) - in the long term.
I just think using YARN to write a new app for everything is a slightly longer path to take if MR can be harmlessly tweaked a bit more to do the same thing along with the other good things it already does.
My concern adding apis/config is that it becomes part of the user interface and I'd like to think through it's implications, and whether it's really necessary, before we commit to it. Makes sense?
Yes, makes sense on the API side. Partly why I went with a simple config-based option on doing this.