[ATLAS-4153] [Atlas: Spooling] The order of the entities created in atlas is not same as the order created in hive

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: atlas-core
• Assignee: Radhika Kundam
• Reporter: Dharshana M Krishnamoorthy
• Created: 11/Feb/21 10:00
• Updated: 11/May/21 01:24
• Resolved: 11/May/21 01:23

Description

Steps to re-produce:

• Stop kafka broker
• Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ... abc_table_n)
• Make sure the data is spooled
• Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ... xyz_table_n)
• Wait for 5 mins for the tables to reflect in atlas
• Fire a basic search with abc_* prefix and verify the tables are created
• Collect the createdTime of all the table and verify the order

Here tables were created with prefix btwxb_table.

Guid: a55ca87c-1122-4ed4-9812-5047cc8a7968 of btwxb_table_6 has "createTime": 1612959902106,

{
    "referredEntities": {...},
    "entity": {
        "typeName": "hive_table",
        "attributes": {
            "owner": "hrt_qa",
            "temporary": false,
            "lastAccessTime": 1612959527000,
            "aliases": null,
            "replicatedTo": null,
            "userDescription": null,
            "replicatedFrom": null,
            "qualifiedName": "default.btwxb_table_6@cm",
            "displayName": null,
            "columns": [
                {
                    "guid": "3cf7b33b-7fb7-4de4-a967-7f89410b3a00",
                    "typeName": "hive_column"
                },
                {
                    "guid": "7c5b72ae-e100-467a-b3b2-caade0b7a7a1",
                    "typeName": "hive_column"
                }
            ],
            "description": null,
            "viewExpandedText": null,
            "tableType": "MANAGED_TABLE",
            "sd": {
                "guid": "7a1e2b04-f786-4f42-a3c4-31a0ad872256",
                "typeName": "hive_storagedesc"
            },
            "createTime": 1612959527000,
            "name": "btwxb_table_6",
            "comment": null,
            "partitionKeys": [],
            "parameters": {...},
            "retention": 0,
            "viewOriginalText": null,
            "db": {
                "guid": "a8a3227f-ee7a-49c2-8f9e-98534a3146eb",
                "typeName": "hive_db"
            }
        },
        "guid": "a55ca87c-1122-4ed4-9812-5047cc8a7968",
        "isIncomplete": false,
        "status": "ACTIVE",
        "createdBy": "hive",
        "updatedBy": "hrt_qa",
        "createTime": 1612959902106,
        "updateTime": 1612959911756,
        "version": 0,
        "relationshipAttributes": {...},
        "labels": []
    }
} 

Guid: 70738a7e-d29f-4234-82e7-adbdc4feaed6 of btwxb_table_1 has "createTime": 1612959903892, 

{
    "referredEntities": {...},
    "entity": {
        "typeName": "hive_table",
        "attributes": {
            "owner": "hrt_qa",
            "temporary": false,
            "lastAccessTime": 1612959523000,
            "aliases": null,
            "replicatedTo": null,
            "userDescription": null,
            "replicatedFrom": null,
            "qualifiedName": "default.btwxb_table_1@cm",
            "displayName": null,
            "columns": [
                {
                    "guid": "c283ee8c-d292-406a-a062-d81b12614636",
                    "typeName": "hive_column"
                },
                {
                    "guid": "cbb19940-e513-4295-a145-5b2d6691ea13",
                    "typeName": "hive_column"
                }
            ],
            "description": null,
            "viewExpandedText": null,
            "tableType": "MANAGED_TABLE",
            "sd": {
                "guid": "69d7f6fc-8194-412f-a723-5b142ade1c31",
                "typeName": "hive_storagedesc"
            },
            "createTime": 1612959523000,
            "name": "btwxb_table_1",
            "comment": null,
            "partitionKeys": [],
            "parameters": {...},
            "retention": 0,
            "viewOriginalText": null,
            "db": {
                "guid": "a8a3227f-ee7a-49c2-8f9e-98534a3146eb",
                "typeName": "hive_db"
            }
        },
        "guid": "70738a7e-d29f-4234-82e7-adbdc4feaed6",
        "isIncomplete": false,
        "status": "ACTIVE",
        "createdBy": "hive",
        "updatedBy": "hrt_qa",
        "createTime": 1612959903892,
        "updateTime": 1612959919910,
        "version": 0,
        "relationshipAttributes": {...},
        "labels": []
    }
} 

 Attributes's create time:

btwxb_table_1 :  1612959523000

btwxb_table_6 : 1612959527000**

In hive btwxb_table_1 is created before btwxb_table_6

Entity's create time:**

btwxb_table_1  has "createTime": 1612959903892, and 

btwxb_table_6 has "createTime": 1612959902106, which is smaller than btwxb_table_1's createTime. 

Which means table_6 is created before table_1 in atlas

 

Issue Links

• links to: Code Review