[NIFI-11814] Controller Services that are enabling (not enabled) appear to block parameter context changes

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.21.0, 1.22.0
• Fix Version/s: 2.0.0-M1, 1.23.0
• Component/s: Core Framework
• Assignee: Mark Payne
• Reporter: Joe Witt
• Created: 14/Jul/23 22:59
• Updated: 19/Jul/23 19:27
• Resolved: 19/Jul/23 19:27

Description

Flagging as a blocker for given the problem it can cause.

Scenario:

• User has a significant number of Controller Services (more than 1000 of them)
• Some Controller Services reference Param Context values some dont.
• A handful of the Controller Services are failing to start but are not disabled. They're invalid but listed as 'enabling' when looking at diagnostics for instance. Such as

  DBCPConnectionPool : 1003 total, {enabling=1, enabled=278, disabled=724}
  

• Now user tries to edit a parameter in a parameter context that does not reference any controller services (or they make a new param context then try to add a param at all). It will timeout after about 30 seconds failing due to

  2023-07-14 14:21:13,791 WARN [Replicate Request Thread-219480] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/process-groups/root/controller-services to host:9443 due to java.net.ProtocolException: unexpected end of stream
  2023-07-14 14:21:13,791 WARN [Replicate Request Thread-219480] o.a.n.c.c.h.r.ThreadPoolRequestReplicator
  java.net.ProtocolException: unexpected end of stream
          at okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.kt:415)
          at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:276)
          at okio.RealBufferedSource.read(RealBufferedSource.kt:189)
          at okio.RealBufferedSource.exhausted(RealBufferedSource.kt:197)
          at okio.InflaterSource.refill(InflaterSource.kt:112)
          at okio.InflaterSource.readOrInflate(InflaterSource.kt:76)
          at okio.InflaterSource.read(InflaterSource.kt:49)
          at okio.GzipSource.read(GzipSource.kt:69)
          at okio.Buffer.writeAll(Buffer.kt:1290)
          at okio.RealBufferedSource.readByteArray(RealBufferedSource.kt:236)
          at okhttp3.ResponseBody.bytes(ResponseBody.kt:124)
          at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.getResponseBytes(OkHttpReplicationClient.java:168)
          at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:138)
          at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:130)
          at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:645)
          at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:869)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
  2023-07-14 14:21:13,791 ERROR [Parameter Context Update Thread-2] o.a.n.web.api.ParameterContextResource Failed to update Parameter Context
  org.apache.nifi.web.util.LifecycleManagementException: Failed while waiting for Controller Services to finish transitioning to a state of DISABLED
          at org.apache.nifi.web.util.ClusterReplicationComponentLifecycle.activateControllerServices(ClusterReplicationComponentLifecycle.java:469)
          at org.apache.nifi.web.util.ParameterUpdateManager.disableControllerServices(ParameterUpdateManager.java:273)
          at org.apache.nifi.web.util.ParameterUpdateManager.updateParameterContexts(ParameterUpdateManager.java:144)
          at org.apache.nifi.web.api.ParameterContextResource.lambda$submitUpdateRequest$17(ParameterContextResource.java:832)
          at org.apache.nifi.web.api.concurrent.AsyncRequestManager$2.run(AsyncRequestManager.java:117)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java
  
  

• Ironically then if you look at why some of the controller services failed to enable it can be for reasons like a parameter they reference not being available. Which you could not go fix as that would then timeout due to the enabling controller service. For example

  2023-07-14 22:21:26,912 ERROR [Timer-Driven Process Thread-4] o.a.n.c.s.StandardControllerServiceNode Unable to enable StandardControllerServiceNode[service=DBCPConnectionPool[id=someuuid], name=ENEL-DBCPConnectionPool, active=true].  Last known validation state was INVALID : ['Database Connection URL' is invalid because Property references Parameter 'hostname' but the currently selected Parameter Context does not have a Parameter with that name, 'Database Connection URL' is invalid because Property references Parameter 'DB_Name' but the currently selected Parameter Context does not have a Parameter with that name, 'Database User' is invalid because Property references Parameter 'DB_User' but the currently selected Parameter Context does not have a Parameter with that name, 'Password' is invalid because Property references Parameter 'DB_Password' but the currently selected Parameter Context does not have a Parameter with that name]
  

Issue Links

• links to: GitHub Pull Request #7493