[YARN-7732] Support Generic AM Simulator from SynthGenerator

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Component/s: scheduler-load-simulator
• Hadoop Flags: Reviewed
• Assignee: Young Chen
• Reporter: Young Chen
• Created: 11/Jan/18 00:34
• Updated: 21/Feb/18 18:33
• Resolved: 21/Feb/18 18:33

Description

Extract the MapReduce specific set-up in the SLSRunner into the MRAMSimulator, and enable support for pluggable AMSimulators.

Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, for example startAMFromSynthGenerator() calls this:

 

runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId,

            jobStartTimeMS, jobFinishTimeMS, containerList, reservationId,

            job.getDeadline(), getAMContainerResource(null));

where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce"

The container set up was also only suitable for mapreduce: 

 

Version:1.0 StartHTML:000000286 EndHTML:000012564 StartFragment:000003634 EndFragment:000012474 StartSelection:000003700 EndSelection:000012464 SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java 
// map tasks

for (int i = 0; i < job.getNumberMaps(); i++) {
	TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0);
	RMNode node =
		nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size())))
			.getNode();
	String hostname = "/" + node.getRackName() + "/" + node.getHostName();
	long containerLifeTime = tai.getRuntime();
	Resource containerResource =
		Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(),
			(int) tai.getTaskInfo().getTaskVCores());
	containerList.add(new ContainerSimulator(containerResource,
		containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map"));
}



// reduce tasks
for (int i = 0; i < job.getNumberReduces(); i++) {
	TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0);
	RMNode node =
		nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size())))
			.getNode();
	String hostname = "/" + node.getRackName() + "/" + node.getHostName();
	long containerLifeTime = tai.getRuntime();
	Resource containerResource =
		Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(),
			(int) tai.getTaskInfo().getTaskVCores());
	containerList.add(
		new ContainerSimulator(containerResource, containerLifeTime,
		hostname, DEFAULT_REDUCER_PRIORITY, "reduce"));
}

 

In addition, the syn.json format supported only mapreduce (the parameters were very specific: mtime, rtime, mtasks, rtasks, etc..).

This patch aims to introduce a new syn.json format that can describe generic jobs, and the SLS setup required to support the synth generation of generic jobs.

See syn_generic.json for an equivalent of the previous syn.json in the new format.

Using the new generic format, we describe a StreamAMSimulator simulates a long running streaming service that maintains N number of containers for the lifetime of the AM. See syn_stream.json.

 

Issue Links

• depends upon: YARN-7798 Refactor SLS Reservation Creation