[HUDI-4994] DatahubSyncTool does not correctly re-ingest soft-deleted entities

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: meta-sync
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Pramod Biligiri
• Created: 07/Oct/22 10:38
• Updated: 12/Oct/22 07:39

Description

Datahub has a notion of soft-deletes (the entity still exists in the database with a status=removed:true). Such entities could get re-ingested with new properties at a later time, such that the older one gets overwritten. The current implementation in DatahubSyncTool does not handle this scenario. It fails to update the status flag to removed:false during ingest, which means the entity won't surface in the Datahub UI at all.

Ref: See sections on Soft Delete and Hard Delete in the Datahub docs: https://datahubproject.io/docs/how/delete-metadata/#soft-delete-the-default

Issue Links

• links to: GitHub Pull Request #6886