[KAFKA-12889] log clean group consider empty log segment to avoid empty log left

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 0.10.1.1, 2.8.0
• Fix Version/s: 3.0.0
• Component/s: log cleaner
• Assignee: Unassigned
• Reporter: qiang Liu
• Created: 04/Jun/21 07:04
• Updated: 22/Jun/21 02:12
• Resolved: 19/Jun/21 22:34

Description

to avoid log index 4 byte relative offset overflow, log cleaner group check log segments offset to make sure group offset range not exceed Int.MaxValue.

this offset check currentlly not cosider next is next log segment is empty, so there will left empty log files every about 2^31 messages.

the left empty logs will be reprocessed every clean cycle, which will rewrite it with same empty content, witch cause little no need io.

for __consumer_offsets topic, normally we can set cleanup.policy to compact,delete to get rid of this.

my cluster is 0.10.1.1, but after aylize trunk code, it should has same problem too.

 

some of my left empty logs,(run ls -l)

rw-r---- 1 u g 0 Dec 16 2017 00000000000000000000.index
rw-r---- 1 u g 0 Dec 16 2017 00000000000000000000.log
rw-r---- 1 u g 0 Dec 16 2017 00000000000000000000.timeindex
rw-r---- 1 u g 0 Jan  15 2018 00000000002148249632.index
rw-r---- 1 u g 0 Jan  15 2018 00000000002148249632.log
rw-r---- 1 u g 0 Jan  15 2018 00000000002148249632.timeindex
rw-r---- 1 u g 0 Jan  27 2018 00000000004295766494.index
rw-r---- 1 u g 0 Jan  27 2018 00000000004295766494.log
rw-r---- 1 u g 0 Jan  27 2018 00000000004295766494.timeindex

 

Issue Links

• links to: GitHub Pull Request #10818