Details
-
New Feature
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
Description
In Data-lake <-> Workload-cluster scenario, versions of the services available on clusters are not always the same. Especially when Kafka versions are different on the clusters, many a time the Kafka client (on workload-cluster) becomes incompatible with the Kafka broker (on data-lake). This leads to failure in the delivery of Atlas Hook notification as the mechanism is dependent on the Kafka client's capability to deliver messages.
To resolve such an issue there is no other way but to make Kafka client compatible with the broker by upgrading or degrading the client or the broker. Often this is not straight-fwd and the easiest thing to do.
We need another mechanism independent of the Kafka client to deliver hook notifications to the Atlas consumer.
There are various REST API available today to perform CURD on Atlas entities. Those APIs process notifications in real-time as they arrive, since event processing is slow and time-consuming these APIs cannot give the throughput which hooks require.
We need another REST interface that is lightweight and does not process messages as they arrive instead, it could dump the messages to the local kafka. Since Kafka will be available locally in the cluster, the interface will never face the problem of version mismatch of client-broker, as faced by service hooks running on a different cluster.
As publishing messages through REST notification is not instantaneous compared to Kafka notification, we need to explicitly handle the ordering of messages published through REST notification. Attached the document explains the new architecture of REST Notification and how it handles out of ordered messages.
Attachments
Attachments
Issue Links
- links to
