[NIFI-10757] Support Kubernetes as Leader Election Manager and State Provider

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0-M1
• Component/s: Core Framework
• Epic Name: Support Kubernetes as Leader Election Manager and State Provider
• Epic Status: To Do
• Assignee: David Handermann
• Reporter: David Handermann
• Created: 03/Nov/22 19:41
• Updated: 19/Oct/23 21:05
• Resolved: 19/Oct/23 21:05

Description

Kubernetes is a common container deployment platform for Apache NiFi, with various efforts available to manage application orchestration.

Current versions of NiFi require ZooKeeper for clustered deployments, leveraging Curator for the leader election implementation, and using ZookKeeper nodes for providing cluster state information. Kubernetes provides standard resources for leader election and state persistence that could serve as alternatives to ZooKeeper. New component implementations should be developed to enable Kubernetes-based clustered deployments without the need for ZooKeeper.

The StateProvider interface is defined in the public nifi-api module, so that component can be provided in an extension NAR. The Kubernetes implementation should leverage ConfigMaps to persist state based for each component identifier requested. ConfigMap size is limited to 1 MB, corresponding to the internal limit of etcd, which also matches the standard limit for ZooKeeper. The implementation will be configurable through the standard state-management.xml configuration.

The LeaderElectionManager interface is defined in nifi-framework-core, providing the contract for implementation. With Leader Election being a fundamental framework concern, this interface is not part of the public nifi-api open for extension. The Kubernetes implementation should incorporate module refactoring to separate the interface from the ZooKeeper and Kubernetes implementations. Several methods on the current ZooKeeper implementation can be abstracted for shared tracking for registration and election operations. The Kubernetes Lease resource should be used to implement a new Leader Election Manager. The Fabric8 and official Kubernetes Java client libraries also support standard abstractions for implementing the leader election pattern. The implementation approach should evaluate the transitive dependencies for client libraries to determine the best option. The new implementation should provide new settings for NiFi application properties to select Kubernetes instead of ZooKeeper for leader election.

In addition to component implementation, new toolkit features should be developed to support migration between ZooKeeper and Kubernetes. The existing ZooKeeper Migrator supports retrieving state information and sending it to a new ZooKeeper cluster, so a similar approach could be implemented for Kubernetes state information.

Implementation should also include documentation updates describing new properties and configuration examples.

Issue Links

• relates to: NIFI-8743 Support for Kubernetes Highly available cluster and ease of use integration

Sub-Tasks

• Add Kubernetes Leader Election Manager and State Provider: Resolved
• Add Migration Capabilities from ZooKeeper to Kubernetes: Resolved