[NIFI-11373] ConsumeAzureEventHub is keep on opening files and leaving it when configured it with wrong eventhub name

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.16.3
• Component/s: NiFi Stateless
• Assignee: Unassigned
• Reporter: Mahendra hegde
• Created: 03/Apr/23 11:45
• Updated: 03/Apr/23 11:45

Description

Hello team,

When ConsumeAzureEventHub processor is configured with invalid 'Event Hub Name' field, it keeps on creating open pipes on machine (I run on linux).

After some time Nifi goes down with "Too many open files" error.

 

(I thin k this issue is there with 'PutAzureEventHub' processor also)

 

Log sample:
ConsumeAzureEventHub[id=e07e4e61-e910-3c0b-5951-d9087c2e6270] An error occurred while receiving messages from Azure Event Hub expirydays at partition 0, consumerGroupName=o9, exception=com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'dan-eu-t-evh800-hub-spo-ingestion:eventhub:expirydays~32766|o9' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions.  TrackingId:9b129ebe-cf90-4542-892c-8dc37f1a7461_B4, SystemTracker:dan-eu-t-evh800-hub-spo-ingestion:eventhub:expirydays~32766|o9, Timestamp:2023-03-28T08:52:17 TrackingId:62dd911a030747feab4a63e58190b9b3_G7, SystemTracker:gateway5, Timestamp:2023-03-28T08:52:17, errorContext[NS: DAN-EU-T-EVH800-Hub-SPO-Ingestion.servicebus.windows.net, PATH: expirydays/ConsumerGroups/o9/Partitions/0, REFERENCE_ID: LN_f52b86_1679993537559_b9b3_G7]: com.microsoft.azure.eventprocessorhost.ExceptionWithAction: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'dan-eu-t-evh800-hub-spo-ingestion:eventhub:expirydays~32766|o9' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions.  TrackingId:9b129ebe-cf90-4542-892c-8dc37f1a7461_B4, SystemTracker:dan-eu-t-evh800-hub-spo-ingestion:eventhub:expirydays~32766|o9, Timestamp:2023-03-28T08:52:17 TrackingId:62dd911a030747feab4a63e58190b9b3_G7, SystemTracker:gateway5, Timestamp:2023-03-28T08:52:17, errorContext[NS: DAN-EU-T-EVH800-Hub-SPO-Ingestion.servicebus.windows.net, PATH: expirydays/ConsumerGroups/o9/Partitions/0, REFERENCE_ID: LN_f52b86_1679993537559_b9b3_G7]

• Caused by: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'dan-eu-t-evh800-hub-spo-ingestion:eventhub:expirydays~32766|o9' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions.  TrackingId:9b129ebe-cf90-4542-892c-8dc37f1a7461_B4, SystemTracker:dan-eu-t-evh800-hub-spo-ingestion:eventhub:expirydays~32766|o9, Timestamp:2023-03-28T08:52:17 TrackingId:62dd911a030747feab4a63e58190b9b3_G7, SystemTracker:gateway5, Timestamp:2023-03-28T08:52:17, errorContext[NS: DAN-EU-T-EVH800-Hub-SPO-Ingestion.servicebus.windows.net, PATH: expirydays/ConsumerGroups/o9/Partitions/0, REFERENCE_ID: LN_f52b86_1679993537559_b9b3_G7]

Error in Nifi log on 'too many open files' :

java.io.FileNotFoundException: /provenance/provenance_rep1/toc/288784036.toc (Too many open files)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
    at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63)
    at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125)
    at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273)
    at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190)
    at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104)
    at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186)
    at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:1006)
    at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:577)
    at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:548)
    at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:501)
    at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
    at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1283)
    at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
    at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
    at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2023-03-28 03:59:19,589 ERROR [Timer-Driven Process Thread-184] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store

 

Thanks

Mahendra