Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3198

update/delete/truncate operations ATLAS_HOOK and ATLAS_ENTITIES improvements

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: trunk
    • Component/s: None
    • Labels:
      None

      Description

      With the recent changes, we have now stopped ingesting simple insert operations since with this operation, there is no change in the metadata. Ideally, we should be doing similar improvements w.r.t. update, delete and truncate operations too. 

      Currently, for update/delete/truncate, we are pushing messages in both ATLAS_HOOK as well as ATLAS_ENTITIES topic, which can be avoided. 

      If we stop ingesting updates/deletes/truncates, it will further help us reduce the burden on Kafka and save CPU cycles

      I have created and attached an excel sheet, with the list of operations and the corresponding messages generated in ATLAS_HOOK and ATLAS_ENTITIES topic, along with suitable comments. 

        Attachments

        1. ATLAS_HOOK and ATLAS_ENTITIES messages.xlsx
          80 kB
          Umesh Padashetty
        2. ATLAS-3198.patch
          5 kB
          Yu-Hsin Chang

          Issue Links

            Activity

              People

              • Assignee:
                yuhsin Yu-Hsin Chang
                Reporter:
                umesh.padashetty Umesh Padashetty
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: