[ATLAS-2729] Audit logging fails for large entities

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0, 2.0.0
• Component/s: atlas-core
• Assignee: Madhan Neethiraj
• Reporter: Ayub Pathan
• Created: 31/May/18 20:27
• Updated: 26/Sep/19 00:38
• Resolved: 05/Jun/18 00:18

Description

Created a table with 4000 columns. Notification message from Hive hooks is about 5mb; and the hook sends the message in compressed format. However, processing of this notification fails in Atlas server, due to the following error:

2018-05-28 06:07:37,817 WARN  - [NotificationHookConsumer thread-0:] ~ audit record too long: entityType=hive_table, guid=4124f580-b98d-473b-ba51-12e2ca2c6cec, size=2891084; maxSize=1048576. entity attribute values not stored in audit (EntityAuditListenerV2:189)
2018-05-28 06:07:37,899 ERROR - [NotificationHookConsumer thread-0:] ~ graph rollback due to exception  (GraphTransactionInterceptor:154)
java.lang.IllegalArgumentException: KeyValue size too large
        at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:952)
        at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:940)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:564)
        at org.apache.atlas.repository.audit.HBaseBasedAuditRepository.putEve

This issue was addressed in earlier releases via ATLAS-1463. The fix needs to be reviewed/updated for recent additions in master (i.e. addition of relationshipAttributes.