[ATLAS-4460] Search API gets deleted partitionKeys(and columns) of Hive table

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Component/s: hive-integration
• Assignee: Unassigned
• Reporter: Solbi Choi
• Created: 22/Oct/21 08:22
• Updated: 22/Oct/21 10:17

Description

Problem

When one of the partitionKeys in hive table deleted, atlas search API still gets all partitionKeys including deleted one.

Adding

"excludeDeletedEntities": True

in request json doesn't work in this situation.

 

Reproduce

• Create hive table with partition key and sync it using hive-import.
• Delete the hive table and re-create hive table with same name but without partition key this time. Re-sync using hive-import.
• Then you can see the partitionKey deleted in Atlas web view.

• But when trying search API to get the hive table entity using

   request = { "typeName": "hive_table", 
               "attributes": [ "db", "name", "partitionKeys" ],
               "excludeDeletedEntities": True,
               "limit": limit,
               "offset": offset
              }

• You get the deleted partitionKey also.

'partitionKeys': [
{'guid': '****', 'typeName': 'hive_column', 'uniqueAttributes': {'qualifiedName': 'foo.test_partition_drop.ds@primary'}}, 
{'guid': '****', 'typeName': 'hive_column', 'uniqueAttributes': {'qualifiedName': 'foo.test_partition_drop.ts@primary'}}
]

 

 

Additionally, this is reproduced within hive columns, too.

After changing column name by using alter table statement, (eg. foo -> bar)

the Search API gives 2 columns(foo and bar) as result of the hive table even with "excludeDeletedEntities" option.

 

 

 

'columns': [
{'guid': '****', 'typeName': 'hive_column', 'uniqueAttributes': {'qualifiedName': 'db_name.test_partition_drop.bar@primary'}}, 
{'guid': '****', 'typeName': 'hive_column', 'uniqueAttributes': {'qualifiedName': 'db_name.test_partition_drop.foo@primary'}}
]