[ATLAS-2064] Compress Hook Messages Posted to Kafka Atlas Topic

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.8-incubating
• Component/s: atlas-intg
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Due: 25/Aug/17
• Created: 18/Aug/17 22:52
• Updated: 17/May/18 05:08
• Resolved: 17/May/18 05:07

Description

Background
Messages posted by hooks to Atlas Kafka topics are JSON format.

Kafka imposes a 1MB limit on the message size.

Occasionally, depending on operations performed, this threshold is reached. This results in messages being dropped.

The entities are thus not reflected in Atlas.

Solution
Applying compression to these message will alleviate the problem.

Issue Links

• duplicates: ATLAS-2075 Support Messages from Hooks of Arbitrarily Large Size
• links to: Code Review