[NIFI-1457] Excessive bulletins and logging when primary node is revoked by NCM.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.4.1
• Fix Version/s: 0.5.0
• Environment: centOS 7
• Assignee: Matt Gilman
• Reporter: Matthew Clarke
• Created: 01/Feb/16 16:12
• Updated: 04/Feb/16 13:54
• Resolved: 02/Feb/16 19:49

Description

I have a 3 node cluster up and running. The current primary node loses connectivity to NCM and eventually becomes disconnected by NCM because of lack of heartbeat. From NCM, the disconnected node is dropped from cluster and a new primary node is manually elected. When network comms are restored between original primary node and NCM, the NCM receives heartbeat messages once again that claim to be from the primary node. The NCM correctly captures this and revokes that nodes primary role status. The problem is that the bulletins stating that the role has been revoked never stop being produced because the original node heartbeats still claim to be from the primary node.

On original Primary Node I see these messages constantly:2016-02-01 16:05:39,635 INFO [Process NCM Request-2] o.a.n.c.p.impl.SocketProtocolListener Received request c5f13d5b-0f09-4fe2-885e-d2d597339491 from ec2-x.x.x.x.compute.amazonaws.com

On NCM I see these messages constantly in app log:
2016-02-01 11:08:25,636 INFO [Process Pending Heartbeats] o.a.n.c.manager.impl.WebClusterManager Node Event: [id=b996c3c0-996c-4072-ba5a-434294d72036, apiAddress=ec2-x.x.x.x.compute.amazonaws.com, apiPort=8443, socketAddress=ec2-x.x.x.x.compute.amazonaws.com, socketPort=10000] – 'Heartbeat indicates node is running as primary node. Revoking primary role because primary role is assigned to a different node.'

The original primary Node is no longer running and "on primary node" processors; however, it appears the heartbeat message is not being updated to reflect that it is not still the primary node.