[SLIDER-1119] Strip off Zookeeper dependency if Yarn Registry is not enabled

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Slider 0.90.2
• Fix Version/s: Slider 1.0.0
• Assignee: shanyu zhao
• Reporter: shanyu zhao
• Created: 11/May/16 00:42
• Updated: 09/Mar/17 05:55

Description

In some Hadoop cluster where Yarn registry is not enabled, we should not use it in Slider. To check if Yarn registry is enabled or not, we can check an existing hadoop configuration (hadoop.registry.rm.enabled).

Today the main use of Yarn registry is to securely publish Slider AM operation URL, so that Slider agent can connect to Slider AM. Without Yarn registry, we can directly pass in the AM operation URL as a command line parameter to containers when launching the container.

Now admittedly without Yarn registry we lose Slider AM HA. This is because when AM restarts, there is no way to tell containers where to find the new AM. In this case AM would restart those containers. This may not be desirable for some usage but is fine with some other usage scenarios.

Detailed changes needed:
1) In SliderAppMaster, disable service/component registration code if Yarn registry is not enabled.
In SliderClient, disable zookeeper node creation/deletion if Yarn registry is not enabled.
This is done in AbstractSliderLaunchedService.
2) In AbstractProviderService and AgentProviderService, do not use Yarn registry if not enabled.
3) In AgentProviderService.buildContainerLaunchContext(), add "--am-ops-url" parameter to the command line, pass in AM Ops URL from cluster info:
String amOpsUrl = getClusterInfoPropertyValue(StatusKeys.INFO_AM_AGENT_OPS_URL);
4) In agent python code, read "--am-ops-url" parameter and use it instead of reading the value from Zookeeper