[ATLAS-3198] update/delete/truncate operations ATLAS_HOOK and ATLAS_ENTITIES improvements - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 2.0.0
Fix Version/s: trunk
Component/s: None
Labels:
None

Description

With the recent changes, we have now stopped ingesting simple insert operations since with this operation, there is no change in the metadata. Ideally, we should be doing similar improvements w.r.t. update, delete and truncate operations too.

Currently, for update/delete/truncate, we are pushing messages in both ATLAS_HOOK as well as ATLAS_ENTITIES topic, which can be avoided.

If we stop ingesting updates/deletes/truncates, it will further help us reduce the burden on Kafka and save CPU cycles

I have created and attached an excel sheet, with the list of operations and the corresponding messages generated in ATLAS_HOOK and ATLAS_ENTITIES topic, along with suitable comments.