Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-1119

Strip off Zookeeper dependency if Yarn Registry is not enabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Slider 0.90.2
    • Fix Version/s: Slider 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      In some Hadoop cluster where Yarn registry is not enabled, we should not use it in Slider. To check if Yarn registry is enabled or not, we can check an existing hadoop configuration (hadoop.registry.rm.enabled).

      Today the main use of Yarn registry is to securely publish Slider AM operation URL, so that Slider agent can connect to Slider AM. Without Yarn registry, we can directly pass in the AM operation URL as a command line parameter to containers when launching the container.

      Now admittedly without Yarn registry we lose Slider AM HA. This is because when AM restarts, there is no way to tell containers where to find the new AM. In this case AM would restart those containers. This may not be desirable for some usage but is fine with some other usage scenarios.

      Detailed changes needed:
      1) In SliderAppMaster, disable service/component registration code if Yarn registry is not enabled.
      In SliderClient, disable zookeeper node creation/deletion if Yarn registry is not enabled.
      This is done in AbstractSliderLaunchedService.
      2) In AbstractProviderService and AgentProviderService, do not use Yarn registry if not enabled.
      3) In AgentProviderService.buildContainerLaunchContext(), add "--am-ops-url" parameter to the command line, pass in AM Ops URL from cluster info:
      String amOpsUrl = getClusterInfoPropertyValue(StatusKeys.INFO_AM_AGENT_OPS_URL);
      4) In agent python code, read "--am-ops-url" parameter and use it instead of reading the value from Zookeeper

        Attachments

        1. SLIDER-1119.patch
          23 kB
          shanyu zhao

          Activity

            People

            • Assignee:
              shanyu shanyu zhao
              Reporter:
              shanyu shanyu zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: