[ATLAS-2883] Export & Import Process: Large Amount of Data Prevents ExportImport Audits from Being Saved

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.2
• Fix Version/s: 0.8.3
• Component/s: atlas-core
• Assignee: Madhan Neethiraj
• Reporter: Ashutosh Mestry
• Due: 26/Sep/18
• Created: 20/Sep/18 16:11
• Updated: 20/Sep/18 23:32
• Resolved: 20/Sep/18 23:32

Description

Background

The new audit mechanism saves contents of Export/Import results to AtlasExportImportAudit. This is an entity within Atlas, that ends up as a vertex within the database.

Steps to Reproduce

• Populate Atlas with hive_db with over 100 tables each with 100 columns.
• Perform an export.

The following is seen in logs:

audit record too long: entityType=__ExportImportAuditEntry, guid=5a5170fa-8f65-4037-a092-1abbaa725b2f, size=2099039; maxSize=1048576. entity attribute values not stored in audit (EntityAuditListener:153)
graph rollback due to exception  (GraphTransactionInterceptor:71)
org.apache.atlas.exception.AtlasBaseException: Failed to notify EntityAuditListener for change CREATE
    at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:183)
    at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.onEntitiesMutated(AtlasEntityChangeNotifier.java:91)
    at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:207)
    at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:215)
    at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
    at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$EnhancerBySpringCGLIB$$59a2a5e8.createOrUpdate(<generated>)
…
Caused by: org.apache.atlas.AtlasException: Unknown datatype {name=long, description=null}
    at org.apache.atlas.typesystem.persistence.StructInstance.setNull(StructInstance.java:276)
    at org.apache.atlas.repository.audit.EntityAuditListener.clearAttributeValues(EntityAuditListener.java:175)
    at org.apache.atlas.repository.audit.EntityAuditListener.getAuditEventDetail(EntityAuditListener.java:158)
    at org.apache.atlas.repository.audit.EntityAuditListener.createEvent(EntityAuditListener.java:133)
    at org.apache.atlas.repository.audit.EntityAuditListener.onEntitiesAdded(EntityAuditListener.java:63)
    at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:172)
    ... 112 more