Apache Whirr (retired)
  1. Apache Whirr (retired)
  2. WHIRR-693

Control order of actions with waves of whirr.instance-templates

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: core
    • Labels:
      None

      Description

      A cluster can be specified with "waves" by supporting an optional suffix on whirr.instance-templates property in order to provide running base-level services (like ZK or a master-master mysql) to another layer of services during the configuration phase. This enables storing cluster-wide information in zookeeper or doing db creation beforehand.

      An example of a two "wave" (avoiding "phase" which already has a meaning) cluster:

      whirr.instance-templates.0=1 zookeeper, 2 zookeeper, 4 noop

      The .0 templates run like normal during "whirr launch-cluster" and then, during the same Whirr run, the .1 template is applied as a modification of the same cluster so that new host instances are not allocated:

      whirr.instance-templates.1=1 my-master, 2 my-gateway, 4 my-worker

      In the second wave, instance provisioning is inhibited, the instance-templates must have the same number of commas and same sequence of leading numbers, and a role is only allowed to appear in one wave.

      Here is another example:

      whirr.instance-templates.0=1 mysql-master+zookeeper, 1 mysql-master+zookeeper,4 noop
      whirr.instance-templates.1=1 hadoop-namenode+hadoop-jobtracker, 1 hbase-master+hadoop-secondarynn,4 hadoop-tasktracker+hadoop-datanode+hbase-regionserver

      In the first wave, the two mysql-masters form a multi-master ensemble which keeps state information about the setup in the Whirr process (much like ZooKeeperCluster.getHosts(cluster) informs services/hbase of the quorum).

      In the second wave, nodes are not allocated (BootstrapClusterAction.doAction() is inhibited if instances already exist), but all phases for LaunchClusterCommand, including beforeBootstrap() and afterBootstrap() callbacks, are executed. If other whirr.cli.command.*Command are run, they would see a combined whirr.instance-templates that works like normal.

      Obviously, there would be no reason to limit this to 2 waves, but I do not expect more than 10 waves to be useful, so the pattern could enable a suffix of "\.[0-9]".

      Instead of WHIRR-221 which aims to specify a single global order for service startup, the wave format has the advantage that it relies on the implicit synchronization barriers at phases already supported by Whirr, the phases still run actions within phases in parallel, and state generated by a wave is available to later waves.

      Conceptually, this merely splits the whirr.instance-templates into waves of (bootstrap,install,configure,start). If no .[0-9] suffixes are present, then Whirr would behave just like normal.

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Paul Baclace
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development