[KAFKA-845] Re-implement shallow iteration on 0.8.1

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 0.8.1
• Component/s: core
• Assignee: Unassigned
• Reporter: Maxime Brugidou
• Created: 03/Apr/13 13:53
• Updated: 14/Sep/18 12:14
• Resolved: 29/Oct/17 09:28

Description

After KAFKA-732 we decided to not support the shallow iteration feature to speed up the 0.8 release.

This severely impacts the performance for heavy mirroring between clusters. The MirrorMaker needs to decompress/recompress the data which actually triple the CPU compression cost in the data flow (initial compression + 2x compression for mirroring).

It would be great to discuss a solution to re-implement the feature correctly.

Issue Links

• relates to: KAFKA-732 MirrorMaker with shallow.iterator.enable=true produces unreadble messages