Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.1.0, 2.0.0
-
None
Description
We wanted to scale kafka consumers for atlas, as we are getting many lineage messages and processing them just with one consumer is not enough.
There is parameter atlas.notification.hook.numthreads to scale consumers in
NotificationHookConsumer.
But the method:
notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
is always returning one element list, which effectively always starts one consumer
List<NotificationConsumer<T>> consumers = Collections.singletonList(kafkaConsumer);
Log incorrectly says that nuber of consumers has been created:
LOG.info("<== KafkaNotification.createConsumers(notificationType={}, numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, autoCommitEnabled)
