[ATLAS-4838] Export/Import : changeMarker is not set to entity's lastupdatetime or its closer timestamp value

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0, 2.3.0
• Component/s: atlas-core
• Assignee: Priyanshi Shah
• Reporter: Sharmadha S
• Created: 12/Mar/24 18:16
• Updated: 15/Apr/24 08:18
• Resolved: 01/Apr/24 09:29

Description

When an hive table entity is exported using fetchType incremental with changeMarker 0 , after exporting , the changeMarker in the export response is not set to recent timestamp.

For example , on exporting iceberg table with fetchType incremental and changeMarker as 0 :

2024-02-08 13:28:43,838 INFO  - [etp891461509-20 - db3f1141-531c-46c0-909a-5e443d4353a2:] ~ export(item=AtlasObjectId{guid='null', typeName='iceberg_table', uniqueAttributes={qualifiedName:default.iceberg_table2@cm}}; matchType=null, fetchType=incremental): found 1 entities: options: {"itemsToExport":[{"typeName":"iceberg_table","uniqueAttributes":{"qualifiedName":"default.iceberg_table2@cm"}}],"options":{"fetchType":"incremental","changeMarker":0},"fetchTypeOptionValue":"incremental","skipLineageOptionValue":false,"changeTokenFromOptions":0} (StartEntityFetchByExportRequest:123)
2024-02-08 13:28:48,697 INFO  - [etp891461509-20 - db3f1141-531c-46c0-909a-5e443d4353a2:] ~ Assigned guid = f5e6342a-dfdf-4d54-a1b3-b24f987c4ecb to newly created entity (DataAccess:85)
2024-02-08 13:28:48,697 INFO  - [etp891461509-20 - db3f1141-531c-46c0-909a-5e443d4353a2:] ~ addAuditEntry: user: admin, source: cm, target: , operation: EXPORT (ExportImportAuditService:168)
2024-02-08 13:28:48,699 INFO  - [etp891461509-20 - db3f1141-531c-46c0-909a-5e443d4353a2:] ~ <== export(user=admin, from=10.19.28.124): status SUCCESS: changeMarker: 1707225183928 (ExportService:107) 

changeMarker from response is 1707225183928 =>  Tuesday, February 6, 2024 6:43:03.928 PM

 

But lastUpdatetime of iceberg table is 

"createTime": 1707398866970 => Thursday, February 8, 2024 6:57:46.970 PM  
"updateTime": 1707398867375. => Thursday, February 8, 2024 6:57:47.375 PM 

Value is set to 2 days back. This is not then an incremental export , as it will fetch all updates from Tuesday, February 6, 2024 6:43:03.928 PM in next export

Issue Links

• links to: Apache review request