[GIRAPH-571] Giraph on YARN could launch a job-local ZK instance from the AppMaster

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: zookeeper
• Assignee: Unassigned
• Reporter: Eli Reisman
• Created: 16/Mar/13 19:24
• Updated: 16/Mar/13 19:24

Description

Once GIRAPH-13 is in, we can think differently about a lot of things if we choose too.

for one thing, we have had problems launching job-local ZK instances. We could (for YARN) move that functionality to the App Master, having it launch a container just for ZK and populating the Configuration's giraph.zkList setting so when the MRv1 ZK manager code sees the Conf, it will think we already have a non-job-local ZK at zkList's host and port, and will just connect instead of starting another local instance, making the whole affair transparent to existing Giraph code.

Not important, but the YARN patch is currently defaulting to only execute jobs with a non-local ZK instance already running, and giraph.zkList populated with its host:port.

Its quite possible when we get our MRv1 job local ZK working again, we can remove this and it will work right out of the box, there's no reason it won't. But managing extraneous services (especially those that hold up the job setup like launching a ZK) is what the YARN AppMaster is all about anyway. i haven't been able to get our local ZK instance to launch outside of test cases for a while now.