Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.7-incubating
-
None
-
None
Description
Drop table and view requires soft deletes. The reason is whenever a table is dropped , it also may have an associated lineage which consists of a hive_process which N input tables and an output table. If the table is dropped, the lineage edge is also dropped resulting in incorrect lineage history.
With soft deletes, the expected behaviour is to changes the table status to "deleted" and when the table is recreated through a create table statement, then create another vertex/entity for that table with the new state. Also, the lineage for this newly recreated table should be a new hive_process and should not reuse the existing entity/vertex even though the hive statement for that process is the same.