[NIFI-5443] Improve cluster configuration for dynamic scaling

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.7.1
• Component/s: Core Framework
• Labels: cluster docker kubernetes rkt scale security
• Assignee: Unassigned
• Reporter: Andy LoPresto
• Created: 18/Jul/18 22:53
• Updated: 10/Feb/23 19:38

Description

Currently, NiFi is designed for static clusters, with frequent references in configuration files to a priori knowledge of node hostnames, ports, etc.

Efforts should be taken to make NiFi easier to dynamically scale. This can involve containerization improvements via Docker/rkt, deployment improvements via Kubernetes, and abstraction of the configuration values needed to stand up the cluster. A node should be able to join the cluster, and, given the correct keystore and truststore, immediately communicate with other existing nodes in the cluster without requiring direct configuration changes to them, or a restart of any node.

• authorizers.xml
• node identities
• permissions (RW on /proxy)
• ZooKeeper configuration
• etc.

Issue Links

• depends upon: NIFI-5398 Identify cluster communication endpoints via combination of hostname and certificate rather than just certificate DN
• Is contained by: NIFI-5458 Improve NiFi TLS and certificate management
• is related to: NIFI-10975 Add Kubernetes Leader Election Manager and State Provider
• relates to: NIFI-5542 Add support for node groups to FileAccessPolicyProvider