[KAFKA-7897] Invalid use of epoch cache with old message format versions

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.1, 2.1.0
• Fix Version/s: 1.1.2, 2.2.0, 2.1.1, 2.0.2
• Assignee: Jason Gustafson
• Reporter: Jason Gustafson
• Created: 05/Feb/19 01:19
• Updated: 09/Apr/19 22:40
• Resolved: 12/Feb/19 19:22

Description

Message format downgrades are not supported, but they generally work as long as broker/clients at least can continue to parse both message formats. After a downgrade, the truncation logic should revert to using the high watermark, but currently we use the existence of any cached epoch as the sole prerequisite in order to leverage OffsetsForLeaderEpoch. This has the effect of causing a massive truncation after startup which causes re-replication.

I think our options to fix this are to either 1) clear the cache when we notice a downgrade, or 2) forbid downgrades and raise an error.

Issue Links

• causes: KAFKA-8012 NullPointerException while truncating at high watermark can crash replica fetcher thread
• is duplicated by: KAFKA-7886 Some partitions are fully truncated during recovery when log.message.format = 0.10.2 & inter.broker.protocol >= 0.11
• relates to: KAFKA-8036 Log dir reassignment on followers fails with FileNotFoundException for the leader epoch cache on leader election
• links to: GitHub Pull Request #6232
• links to: GitHub Pull Request #6253