[ATLAS-4164] [Atlas: Spooling] Tables created after spooling are created before the spooled tables when there is multiple frequent restart in kafka brokers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 3.0.0
• Component/s: atlas-core
• Labels: spool
• Assignee: Radhika Kundam
• Reporter: Dharshana M Krishnamoorthy
• Created: 18/Feb/21 05:38
• Updated: 03/Aug/21 05:59
• Resolved: 12/May/21 17:16

Description

Scenario:

• Stop kafka broker
• Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ... abc_table_n)
• Make sure the data is spooled
• Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ... xyz_table_n)
• Wait for 5 mins for the tables to reflect in atlas

In this case we expect all the abc_table_* to be created before xyz_table_1, meaning all the spooled tables are created before the tables that are created after spooling.

 

Observation:

createTime of some spooled tables is greater than the create time of the xyz_table_1

 

Sample data:

createTime for tables that are spooled:

[1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253, 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629, 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947, 1613573521737, 1613573514066, 1613573514555, 1613573515065, 1613573515605]

createTime for tables that are created after spooling:

[1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356, 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659, 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004, 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202] 

We expect all spooled tables to have createTime smaller than the tables created after spooling.

But 1613573543253 (Spooled tabled create time) is greater than 1613573540582 (table created after spooling)
which means, the table created after spooling is created before spooled table

Issue Links

• links to: Code Review