Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.2.0
-
None
-
None
Description
Problem
When one of the partitionKeys in hive table deleted, atlas search API still gets all partitionKeys including deleted one.
Adding
"excludeDeletedEntities": True
in request json doesn't work in this situation.
Reproduce
- Create hive table with partition key and sync it using hive-import.
- Delete the hive table and re-create hive table with same name but without partition key this time. Re-sync using hive-import.
- Then you can see the partitionKey deleted in Atlas web view.
- But when trying search API to get the hive table entity using
request = { "typeName": "hive_table", "attributes": [ "db", "name", "partitionKeys" ], "excludeDeletedEntities": True, "limit": limit, "offset": offset }
- You get the deleted partitionKey also.
'partitionKeys': [ {'guid': '****', 'typeName': 'hive_column', 'uniqueAttributes': {'qualifiedName': 'foo.test_partition_drop.ds@primary'}}, {'guid': '****', 'typeName': 'hive_column', 'uniqueAttributes': {'qualifiedName': 'foo.test_partition_drop.ts@primary'}} ]
Additionally, this is reproduced within hive columns, too.
After changing column name by using alter table statement, (eg. foo -> bar)
the Search API gives 2 columns(foo and bar) as result of the hive table even with "excludeDeletedEntities" option.
'columns': [ {'guid': '****', 'typeName': 'hive_column', 'uniqueAttributes': {'qualifiedName': 'db_name.test_partition_drop.bar@primary'}}, {'guid': '****', 'typeName': 'hive_column', 'uniqueAttributes': {'qualifiedName': 'db_name.test_partition_drop.foo@primary'}} ]