[ATLAS-4256] Deferred actions : When failover happens , the deferred tasks are set to COMPLETED without getting executed - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.1.0
Fix Version/s: 3.0.0, 2.2.0
Component/s: atlas-core
Labels:
None

Description

Atlas is running on server1 and server2 and server1 is currently ACTIVE.

Created a 2000 level lineage and added a tag to start of the lineage. Deferred task for tag propagation started.

Stopped server1 , now server2 became ACTIVE.

server1 threw following exception :

2021-04-20 20:07:21,137 ERROR - [atlas-task-0-etp1479696465-120 - 541aca55-2402-43ad-911b-6756d9899b12:] ~ Error executing task. Please perform the operation again! (TaskExecutor$TaskLogger:178)2021-04-20 20:07:21,137 ERROR - [atlas-task-0-etp1479696465-120 - 541aca55-2402-43ad-911b-6756d9899b12:] ~ Error executing task. Please perform the operation again! (TaskExecutor$TaskLogger:178)java.lang.IllegalStateException: Graph has been closed at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph.getAutoStartTx(JanusGraphBlueprintsGraph.java:76) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph.query(JanusGraphBlueprintsGraph.java:176) at org.apache.atlas.repository.graphdb.janus.query.NativeJanusGraphQuery.<init>(NativeJanusGraphQuery.java:59) at org.apache.atlas.repository.graphdb.janus.query.AtlasJanusGraphQuery.createNativeTinkerpopQuery(AtlasJanusGraphQuery.java:54) at org.apache.atlas.repository.graphdb.tinkerpop.query.expr.AndCondition.create(AndCondition.java:85) at org.apache.atlas.repository.graphdb.tinkerpop.query.TinkerpopGraphQuery.vertices(TinkerpopGraphQuery.java:136) at org.apache.atlas.tasks.TaskRegistry.getVertex(TaskRegistry.java:140) at org.apache.atlas.tasks.TaskExecutor$TaskConsumer.run(TaskExecutor.java:91) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

Now server2 /api/atlas/admin/tasks showed the PENDING tasks.

Started server1 now.

Now all PENDING tasks are marked COMPLETED without getting executed.

2021-04-20 20:09:35,450 INFO  - [main:] ~ TaskManagement: Found: 3: Tasks in pending state. (TaskManagement:195)
2021-04-20 20:09:35,500 INFO  - [main:] ~ {"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"81418bdb-5076-458a-abbd-7894d43d0408","createdBy":"hrt_qa","createdTime":1618949204339,"updatedTime":1618949204339,"parameters":{"relationshipGuid":null,"entityGuid":"1231b504-f6b7-45df-829b-b430a9e7c0d6","classificationVertexId":"4272"},"attemptCount":0,"status":"PENDING"} (TaskExecutor$TaskLogger:170)
2021-04-20 20:09:35,503 INFO  - [main:] ~ {"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"1130d28f-e2a6-49a6-ad99-7642042681a3","createdBy":"hrt_qa","createdTime":1618949205136,"updatedTime":1618949205136,"parameters":{"relationshipGuid":null,"entityGuid":"4c2db030-52fe-419c-aa79-c37db0908502","classificationVertexId":"81924136"},"attemptCount":0,"status":"PENDING"} (TaskExecutor$TaskLogger:170)
2021-04-20 20:09:35,503 INFO  - [main:] ~ {"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"dc4e5057-8e13-4234-9ba5-91e18d54a24d","createdBy":"hrt_qa","createdTime":1618949206861,"updatedTime":1618949206861,"parameters":{"relationshipGuid":null,"entityGuid":"c9f543a6-033c-45c7-ab23-a5476b6fad9c","classificationVertexId":"40964200"},"attemptCount":0,"status":"PENDING"} (TaskExecutor$TaskLogger:170)
2021-04-20 20:09:35,532 INFO  - [atlas-task-0-main:] ~ GraphTransaction intercept for org.apache.atlas.repository.store.graph.v2.EntityGraphMapper.propagateClassification (GraphTransactionAdvisor$1:41)
2021-04-20 20:09:35,804 INFO  - [main:] ~ Atlas is in HA Mode, enabling ActiveServerFilter (AtlasSecurityConfig:167)
2021-04-20 20:09:36,203 INFO  - [atlas-task-0-main:] ~ {"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"81418bdb-5076-458a-abbd-7894d43d0408","createdBy":"hrt_qa","createdTime":1618949204339,"updatedTime":1618949204339,"endTime":1618949375617,"parameters":{"relationshipGuid":null,"entityGuid":"1231b504-f6b7-45df-829b-b430a9e7c0d6","classificationVertexId":"4272"},"attemptCount":0,"status":"COMPLETE"} (TaskExecutor$TaskLogger:170)
2021-04-20 20:09:36,306 INFO  - [main:] ~ AuditFilter initialization started (AuditFilter:64)
2021-04-20 20:09:36,307 INFO  - [main:] ~ REST_API_ENABLE_DELETE_TYPE_OVERRIDE=false (AuditFilter:69)
2021-04-20 20:09:36,855 INFO  - [atlas-task-0-main:] ~ {"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"1130d28f-e2a6-49a6-ad99-7642042681a3","createdBy":"hrt_qa","createdTime":1618949205136,"updatedTime":1618949205136,"endTime":1618949376244,"parameters":{"relationshipGuid":null,"entityGuid":"4c2db030-52fe-419c-aa79-c37db0908502","classificationVertexId":"81924136"},"attemptCount":0,"status":"COMPLETE"} (TaskExecutor$TaskLogger:170)
2021-04-20 20:09:37,093 INFO  - [atlas-task-0-main:] ~ {"type":"CLASSIFICATION_PROPAGATION_ADD","guid":"dc4e5057-8e13-4234-9ba5-91e18d54a24d","createdBy":"hrt_qa","createdTime":1618949206861,"updatedTime":1618949206861,"endTime":1618949376888,"parameters":{"relationshipGuid":null,"entityGuid":"c9f543a6-033c-45c7-ab23-a5476b6fad9c","classificationVertexId":"40964200"},"attemptCount":0,"status":"COMPLETE"} (TaskExecutor$TaskLogger:170)