[ATLAS-4480] [API] Observing slowness in rest api response

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: atlas-core
• Assignee: Unassigned
• Reporter: Dharshana M Krishnamoorthy
• Created: 15/Nov/21 10:52
• Updated: 15/Nov/21 10:52

Description

Observing slowness in the api response

Eg: Scenario 

1. Create a entity
2. Soft delete it
3. Purge the entity
4. Verify purge audit

Here we fire a admin api call immediately after the entity is purged

But the admin api response is empty []

If we query after some time, we can see the audit is present for the purge call

Logs showing the above steps 

2021-11-12 10:34:17,702|INFO|MainThread|atlasv2.py:659 - get_entity_def()|https://quasar-gmxrxz-2.quasar-gmxrxz.root.hwx.site:31443/api/atlas/v2/entity/guid/e15aeb8e-cfb2-4b9d-96f1-b9bce94586e4
2021-11-12 10:34:17,703|INFO|MainThread|atlas.py:1135 - http_get_request()|https://quasar-gmxrxz-2.quasar-gmxrxz.root.hwx.site:31443/api/atlas/v2/entity/guid/e15aeb8e-cfb2-4b9d-96f1-b9bce94586e4
2021-11-12 10:34:17,703|INFO|MainThread|atlas.py:1202 - http_request()|https://quasar-gmxrxz-2.quasar-gmxrxz.root.hwx.site:31443/api/atlas/v2/entity/guid/e15aeb8e-cfb2-4b9d-96f1-b9bce94586e4
2021-11-12 10:34:17,704|INFO|MainThread|atlas.py:1209 - http_request()|HTTP Method: GET, Body: None
2021-11-12 10:34:17,705|INFO|MainThread|atlas.py:1238 - http_request()|Making HTTP requests via Kerberos auth
2021-11-12 10:34:17,787|INFO|MainThread|atlas.py:1257 - http_request()|HTTP response code: 200
2021-11-12 10:34:17,790|INFO|MainThread|atlas.py:807 - set_base_url()|Base url: https://quasar-gmxrxz-2.quasar-gmxrxz.root.hwx.site:31443
2021-11-12 10:34:17,791|INFO|MainThread|atlas.py:1202 - http_request()|https://quasar-gmxrxz-2.quasar-gmxrxz.root.hwx.site:31443/api/atlas/admin/purge
2021-11-12 10:34:17,791|INFO|MainThread|atlas.py:1209 - http_request()|HTTP Method: PUT, Body: ["e15aeb8e-cfb2-4b9d-96f1-b9bce94586e4"]
2021-11-12 10:34:17,792|INFO|MainThread|atlas.py:1238 - http_request()|Making HTTP requests via Kerberos auth
2021-11-12 10:34:18,184|INFO|MainThread|atlas.py:1257 - http_request()|HTTP response code: 200
2021-11-12 10:34:18,186|INFO|MainThread|utils.py:18 - purge_entity_and_verify_status()|{u'mutatedEntities': {u'PURGE': [{u'status': u'DELETED', u'isIncomplete': False, u'guid': u'9c9d8fc0-3f81-4d24-987f-60172e776d84', u'classifications': [], u'labels': [], u'typeName': u'hive_column', u'meaningNames': [], u'displayText': u'id', u'meanings': [], u'attributes': {u'owner': u'hrt_qa', u'qualifiedName': u'default.table_lxjgr.id@cm', u'name': u'id'}, u'classificationNames': []}, {u'status': u'DELETED', u'isIncomplete': False, u'guid': u'6c7b8038-3c46-4d25-9ef3-c4bdb7ddb5a3', u'classifications': [], u'labels': [], u'typeName': u'hive_column', u'meaningNames': [], u'displayText': u'name', u'meanings': [], u'attributes': {u'owner': u'hrt_qa', u'qualifiedName': u'default.table_lxjgr.name@cm', u'name': u'name'}, u'classificationNames': []}, {u'status': u'DELETED', u'isIncomplete': False, u'guid': u'e15aeb8e-cfb2-4b9d-96f1-b9bce94586e4', u'classifications': [], u'labels': [], u'typeName': u'hive_table', u'meaningNames': [], u'displayText': u'table_lxjgr', u'meanings': [], u'attributes': {u'owner': u'hrt_qa', u'qualifiedName': u'default.table_lxjgr@cm', u'createTime': 1636713248000, u'name': u'table_lxjgr'}, u'classificationNames': []}, {u'status': u'DELETED', u'isIncomplete': False, u'guid': u'6132f1b4-6efb-45af-8ec2-70f197aa16c2', u'classifications': [], u'labels': [], u'typeName': u'hive_storagedesc', u'meaningNames': [], u'displayText': u'default.table_lxjgr@cm_storage', u'meanings': [], u'attributes': {u'qualifiedName': u'default.table_lxjgr@cm_storage'}, u'classificationNames': []}, {u'status': u'DELETED', u'isIncomplete': False, u'guid': u'0354d287-c598-4325-b0ec-d48dfaf6fd9a', u'classifications': [], u'labels': [], u'typeName': u'hive_table_ddl', u'meaningNames': [], u'displayText': u'default.table_lxjgr@cm:1636713248787', u'meanings': [], u'attributes': {u'qualifiedName': u'default.table_lxjgr@cm:1636713248787'}, u'classificationNames': []}]}}
2021-11-12 10:34:18,186|INFO|MainThread|utils.py:72 - purge_async_deleted_entities()|{u'mutatedEntities': {u'PURGE': [{u'status': u'DELETED', u'isIncomplete': False, u'guid': u'9c9d8fc0-3f81-4d24-987f-60172e776d84', u'classifications': [], u'labels': [], u'typeName': u'hive_column', u'meaningNames': [], u'displayText': u'id', u'meanings': [], u'attributes': {u'owner': u'hrt_qa', u'qualifiedName': u'default.table_lxjgr.id@cm', u'name': u'id'}, u'classificationNames': []}, {u'status': u'DELETED', u'isIncomplete': False, u'guid': u'6c7b8038-3c46-4d25-9ef3-c4bdb7ddb5a3', u'classifications': [], u'labels': [], u'typeName': u'hive_column', u'meaningNames': [], u'displayText': u'name', u'meanings': [], u'attributes': {u'owner': u'hrt_qa', u'qualifiedName': u'default.table_lxjgr.name@cm', u'name': u'name'}, u'classificationNames': []}, {u'status': u'DELETED', u'isIncomplete': False, u'guid': u'e15aeb8e-cfb2-4b9d-96f1-b9bce94586e4', u'classifications': [], u'labels': [], u'typeName': u'hive_table', u'meaningNames': [], u'displayText': u'table_lxjgr', u'meanings': [], u'attributes': {u'owner': u'hrt_qa', u'qualifiedName': u'default.table_lxjgr@cm', u'createTime': 1636713248000, u'name': u'table_lxjgr'}, u'classificationNames': []}, {u'status': u'DELETED', u'isIncomplete': False, u'guid': u'6132f1b4-6efb-45af-8ec2-70f197aa16c2', u'classifications': [], u'labels': [], u'typeName': u'hive_storagedesc', u'meaningNames': [], u'displayText': u'default.table_lxjgr@cm_storage', u'meanings': [], u'attributes': {u'qualifiedName': u'default.table_lxjgr@cm_storage'}, u'classificationNames': []}, {u'status': u'DELETED', u'isIncomplete': False, u'guid': u'0354d287-c598-4325-b0ec-d48dfaf6fd9a', u'classifications': [], u'labels': [], u'typeName': u'hive_table_ddl', u'meaningNames': [], u'displayText': u'default.table_lxjgr@cm:1636713248787', u'meanings': [], u'attributes': {u'qualifiedName': u'default.table_lxjgr@cm:1636713248787'}, u'classificationNames': []}]}}
2021-11-12 10:34:18,187|INFO|MainThread|atlas.py:807 - set_base_url()|Base url: https://quasar-gmxrxz-2.quasar-gmxrxz.root.hwx.site:31443
2021-11-12 10:34:18,190|INFO|MainThread|atlas.py:1202 - http_request()|https://quasar-gmxrxz-2.quasar-gmxrxz.root.hwx.site:31443/api/atlas/admin/audits
2021-11-12 10:34:18,190|INFO|MainThread|atlas.py:1209 - http_request()|HTTP Method: POST, Body: {"sortOrder": "DESCENDING", "limit": 1, "sortBy": "startTime", "auditFilters": {"criterion": [{"operator": "like", "attributeName": "userName", "attributeValue": "hrt_qa"}, {"operator": "like", "attributeName": "operation", "attributeValue": "PURGE"}], "condition": "AND"}, "offset": 0}
2021-11-12 10:34:18,191|INFO|MainThread|atlas.py:1238 - http_request()|Making HTTP requests via Kerberos auth
2021-11-12 10:34:18,259|INFO|MainThread|atlas.py:1257 - http_request()|HTTP response code: 200
2021-11-12 10:34:18,260|INFO|MainThread|utils.py:26 - verify_purge_audit()|[] 

Please Note the [] in the last line. The response is empty

This is due to the slowness and the data is present if we query after 2minutes

cc: sharmadhas 

This example is just one of such occurrence, this is seen in many scenarios