[ATLAS-492] Hive Hook Improvements

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 0.7-incubating
• Assignee: Unassigned
• Reporter: Erik Bergenholtz
• Created: 16/Feb/16 14:37
• Updated: 23/Jul/21 06:22
• Resolved: 18/May/20 02:44

Description

Requirements:

• Support ALTER statements (add/replace/rename cols)
• Support DROP statements ( support drop database, table, view)
• Support LOAD statements
• Remodelling changes

Issue Links

• contains: ATLAS-653 insert into local dir, load from local dir does not show up in lineage
• contains: ATLAS-876 No lineage for a re-created ctas table
• contains: ATLAS-877 CreateTime of an entity(for example hive table) changes for every update
• contains: ATLAS-619 Queries should be normalized when tracking lineage at table
• contains: ATLAS-635 Process showing old entity name where as actual entity is renamed.
• contains: ATLAS-636 Tags/Traits associated with partitionkeys should be retained after rename table.
• contains: ATLAS-652 Delete on a non-existing table/view/db will result in 404
• is related to: ATLAS-782 Hive Hook : Append to list of recentQueries instead of replacing
• is related to: ATLAS-725 UI : Filter out or highlight deleted entities in search result outputs(dsl, text) , schema view, and lineage graph
• is related to: ATLAS-596 UI : Bucket cols and sortCols within storage descriptor are not displayed correctly
• is related to: ATLAS-685 UI : HDFS datasets are not displayed correctly
• is related to: ATLAS-724 UI: Lineage view should display datasets as the source and target view ( more intuitive )
• is related to: ATLAS-726 UI : dataset lineage process should be sorted according to process start time
• is related to: ATLAS-752 Column renames should retain traits/tags
• is related to: ATLAS-804 For hive_db, "locationUri" field is renamed to "location" but the hive data model json is still showing as "locationUri"
• is related to: ATLAS-638 UI : Recognize and display HDFS Paths with a different notation
• relates to: ATLAS-702 "ALTER TABLE .. NOT STORED AS DIRECTORIES" metadata update is not sent to Atlas.
• relates to: ATLAS-648 Alter view rename does not work across databases
• relates to: ATLAS-758 hdfs location of hive table is pointing to old location even after rename.
• relates to: ATLAS-760 Missing table parameters(last_modified_by and last_modified_time)
• relates to: ATLAS-626 Hive temporary table metadata is captured in atlas.
• relates to: ATLAS-627 Change column name should retain traits/tags assigned to columns
• relates to: ATLAS-636 Tags/Traits associated with partitionkeys should be retained after rename table.
• relates to: ATLAS-641 Lineage for a view created from a view seems to be confusing(from user's perspective).
• relates to: ATLAS-703 No entity notification updates are sent for hive table protection feature. Seems like table protection changes are not yet modelled.
• relates to: ATLAS-772 Ordering of columns is not maintained in schema query response, where as hive table entity response maintains the ordering

Sub-Tasks

• Support Alter Table column commands: Resolved
• Support Alter table commands: Resolved
• Support alter view: Resolved
• Support alter database: Resolved
• Drop support for partitions, select query lineage, roles, principals, resource, hive_type...: Resolved
• Verify insert into select results in table lineage: Resolved
• Support lineage for load table, import, export: Resolved
• support drop table, view: Resolved
• support drop database: Resolved
• Add table information to column class: Resolved
• Remove isComposite on storageDescriptor.cols: Resolved
• Change Data types of all timestamps in Hive model(currently long): Resolved
• Support drop database with cascade: Resolved
• Verify free text search is able to surface search on table parameters: Resolved
• Support delete cascade efficiently: Resolved
• Make qualifiedName and name consistent across all Datasets and Process: Resolved
• alter table rename should modify the list of columns and storage descriptor: Resolved
• Parallelize Hive hook operations: Open
• Rename table should retain traits/tags assigned to columns/storage descriptors: Resolved
• alter table rename doesnt work across databases: Resolved
• 'Description/Comment' attribute needs to be normalized hive_db, hive_table: Resolved
• Handle Cluster renaming: Open
• Change from Hive thrift interface to Hive MetastoreClient: Resolved
• import-hive should create the lineage for external tables: Resolved
• Truncate table, insert into local dir, load from local dir , update, delete commands should be easily trackable from a table: Open
• ALTER TABLE TOUCH is not modelled: Open
• "ALTER TABLE .. NOT STORED AS DIRECTORIES" metadata update is not sent to Atlas.: Open
• insert into local dir, load from local dir does not show up in lineage: Resolved