Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5663

FlowFile load balancing keeps re-partitioning

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.8.0
    • Component/s: Core Framework
    • Labels:
      None

      Description

      Scenario

      1. Start a brand-new cluster with only 1 node (nifi0). Using existing multi-node clusters don't reproduce the issue.
      2. Create GenerateFlowFile -> LogAttribute
      3. Then set 'Partition by attribute' LB strategy at the connection
      4. Add 2nd node, nifi1
      5. Generate some FlowFiles. Then load-balance activity never finishes.

      With a 2-node cluster, for some reason, each node ended up having different queuePartitions order at SocketLoadBalancedFlowFileQueue. By adding debug logs, I found each node has followings:

      • nifi0
        • queuePartitions[0} = RemoteQueuePartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b, nodeId=nifi1.example.com:8080]
        • queuePartitions[1} = SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b]
      • nifi1
        • queuePartitions[0} = RemoteQueuePartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b, nodeId=nifi0.example.com:8080]
        • queuePartitions[1} = SwappablePriorityQueueLocalPartition[queueId=14ac9634-0166-1000-ffff-ffffd9ae7f4b]

      Because of this, 'Partition by attribute' LB strategy keeps re-partitioning received FlowFiles between each other in case the calculated attribute value hash points to queuePartitions[0]. Following log is written endlessly:

      2018-10-05 07:09:32,372 DEBUG [Load Balance Server Thread-3] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Received the following FlowFiles from Peer: ...offset=7452, length=180],offset=162,name=10653317458635,size=18]]. Will re-partition FlowFiles to ensure proper balancing across the cluster.
      

      SocketLoadBalancedFlowFileQueue maintains queuePartitions by listening to cluster topology change using ClusterTopologyEventListener. SocketLoadBalancedFlowFileQueueClusterEventListener.onNodeAdded debug log shows the array was empty when the 2nd node (nifi1) is added:

      ClusterEventListener.onNodeAdded. 2018-10-05 07:06:42,883 DEBUG [Process Cluster Protocol Request-10] o.a.n.c.q.c.SocketLoadBalancedFlowFileQueue Node Identifier nifi1.example.com:8080 added to cluster. Node ID's changing from [] to [nifi1.example.com:8080]
      

        Attachments

          Activity

            People

            • Assignee:
              markap14 Mark Payne
              Reporter:
              ijokarumawak Koji Kawamura

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment