[ATLAS-2075] Support Messages from Hooks of Arbitrarily Large Size

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8-incubating
• Fix Version/s: 0.8.2, 1.0.0
• Component/s: atlas-core
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Due: 31/Aug/17
• Created: 21/Aug/17 18:17
• Updated: 17/May/18 05:07
• Resolved: 01/Oct/17 01:21

Description

Background
Messages produced by Hooks have potential to be larger than 1MB, which is the size threshold imposed by Kafka.

Although, compressing the messages (see ATLAS-2064) alleviates the the problem, it is not a complete solution. It is possible even for compressed messages to exceed the size threshold.

Solution
If the compressed message produced exceeds the size threshold, split the messages. Accumulate the message at consumer end.

Account for cases such as:

• Messages are not received in order they are produced.
• Atlas server is shutdown before it can consume all the split messages.

Issue Links

• is duplicated by: ATLAS-2064 Compress Hook Messages Posted to Kafka Atlas Topic
• links to: Code Review