[NIFI-2444] PublishKafka does not generate bulletins when there are problems

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.0.0, 1.0.0-Beta
• Component/s: Extensions
• Labels: beginner newbie
• Assignee: Oleg Zhurakousky
• Reporter: Mark Payne
• Created: 01/Aug/16 00:33
• Updated: 04/Aug/16 21:24
• Resolved: 03/Aug/16 01:06

Description

I noticed extremely poor performance when trying to send data to Kafka via PublishKafka. I saw no bulletins so didn't realize there were problems until I looked into the logs and saw a lot of stack traces:

2016-08-01 00:30:03,749 ERROR [Timer-Driven Process Thread-44] o.a.n.p.kafka.pubsub.KafkaPublisher Failed while waiting for acks from Kafka
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [nifi-provenance]
	at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:686) ~[kafka-clients-0.9.0.1.jar:na]
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:449) ~[kafka-clients-0.9.0.1.jar:na]
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:339) ~[kafka-clients-0.9.0.1.jar:na]
	at org.apache.nifi.processors.kafka.pubsub.KafkaPublisher.publish(KafkaPublisher.java:115) ~[nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.processors.kafka.pubsub.PublishKafka$1.process(PublishKafka.java:248) [nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1880) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.processors.kafka.pubsub.PublishKafka.doRendezvousWithKafka(PublishKafka.java:244) [nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.processors.kafka.pubsub.PublishKafka.rendezvousWithKafka(PublishKafka.java:205) [nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.processors.kafka.pubsub.AbstractKafkaProcessor.onTrigger(AbstractKafkaProcessor.java:192) [nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [nifi-provenance]

It appears that the processor creates a KafkaPublisher class (and perhaps others?) that have their own loggers. These loggers need to be removed and the Processor's logger needs to be passed in so that all log messages generate the appropriate bulletins.

Issue Links

• links to: GitHub Pull Request #758
• links to: GitHub Pull Request #761