Uploaded image for project: 'REEF (Retired)'
  1. REEF (Retired)
  2. REEF-936 Support scale-down in REEF
  3. REEF-1383

Improve MultiRuntimeConfigurationBuilder API

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • REEF
    • None

    Description

      I had another look at MultiRuntimeConfigurationBuilder. It now seems very counter-intuitive to me. Look at the code in HelloREEFMultiYarn.getHybridYarnSubmissionRuntimeConfiguration:

      return new MultiRuntimeConfigurationBuilder()
                  .setDefaultRuntime(org.apache.reef.runtime.yarn.driver.RuntimeIdentifier.RUNTIME_NAME)
                  .setSubmissionRuntime(org.apache.reef.runtime.yarn.driver.RuntimeIdentifier.RUNTIME_NAME)
                  .addRuntime(org.apache.reef.runtime.local.driver.RuntimeIdentifier.RUNTIME_NAME)
                  .addRuntime(org.apache.reef.runtime.yarn.driver.RuntimeIdentifier.RUNTIME_NAME)
                  .setMaxEvaluatorsNumberForLocalRuntime(1)
                  .build();
      

      At no point does the user actually configure the runtimes in question. For instance, what if they want to submit to a YARN instance which needs a correct user name to work? There doesn't seem to be a way to specify that. What if I want to support Mesos next? This tight coupling of the multi-runtime with the (small) set of supported participants creates an explosion in the complexity of all the code associated with this change. What prevents us from creating an API that makes the client code look more like this:

      return new MultiRuntimeConfigurationBuilder()
        .addSubmissionRuntime(YARN, YarnClientConfiguration.CONF.build())
        .addDefaultRuntime(YARN, YARNDriverConfiguration.CONF.build())
        .addRuntime(LOCAL, LocalDriverConfiguration.CONF.set(NUMBER_OF_EVALUATORS, 1).build())
        .build();
      

      This would allow an arbitrary set of runtimes to take part in the multi-runtime. Also, it would remove all special-casing in the multi-runtime for specific (combinations of) runtimes. This should simplify the code in the multi-runtime tremendously.

      On the flip side, runtimes now need to implement those APIs, namely ConfigurationBuilder}}s for the Driver side and the Client side separately. That should be doable and has been introduced via the {{Extensible* builders done as part of this work.

      WDYT?

      Attachments

        Activity

          People

            Unassigned Unassigned
            markus.weimer Markus Weimer
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated: