Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-540 Revamp NiFi Clustering
  3. NIFI-338

The NCM should offer high availability

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Core Framework
    • Labels:
      None

      Description

      Purtell:
      ​I'm separately curious about the level of effort it
      would require to introduce multiple masters.​ I think of like how HBase
      does it: there is only ever one active master but a user can deploy
      multiple standbys to take over service should the active master fail.

      Witt:
      So it is true that we have a single master model today. But that
      single master is solely for command/control of changes to the dataflow
      configuration and is a very lightweight process that does nothing more
      than that. If the master dies then all nodes continue to do what they
      were doing and even site-to-site continues to distribute data. It
      just does so without updates on current loading across the cluster.
      Once the master is brought back on-line then the real-time command and
      control functions return. Building support for a back-up master to
      offer HA of even the command/control side would probably also be a
      considerable effort. This one I'd be curious to hear of cases where
      it was critical to make this part HA.

      Purtell:
      Yes but imagine a NiFi installation, perhaps a hosted service built on top
      of it, where DataFlow Managers expect the command and control aspect of the
      system to be as robust and available as flow processing itself. If one or
      more standby masters are waiting in the wings to take over service for the
      failed active master then automated and unattended failover would be
      possible, and likely to narrow the interval where administrative changes
      may fail.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              apurtell Andrew Purtell
            • Votes:
              3 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: