[KAFKA-7092] Multiple topics (including __consumer_offsets) have not been cleaned for a few months

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.10.2.1
• Component/s: core
• Environment: linux ; kafka 0.10.2.1
• Assignee: Unassigned
• Reporter: Noam Berman
• Created: 24/Jun/18 15:03
• Updated: 25/Jun/18 17:05

Description

Hi,

We stumbled upon a scenario - our 8 node cluster (0.10.2.1) hasn't been cleaning logs for many topics since 26/3/2018 (during which the brokers were restarted). This includes __consumer_offsets, which has since grown to enormouse proportions.

The cluster is an upgraded cluster from 0.10.1.0.

I believe this is a reproduction of either https://issues.apache.org/jira/browse/KAFKA-5413 or https://issues.apache.org/jira/browse/KAFKA-6264, but i haven't found any workaround for this.

output of one of __consumer_offsets partition data folders:

user@kafka-broker0a:/var/lib/kafka/data/__consumer_offsets-100$ ls -ltr
total 132832
-rw-r--r-- 1 kafka kafka 0 Mar 26 07:10 00000000000000000000.log
-rw-r--r-- 1 kafka kafka 320648 Mar 26 07:11 00000000004729570626.log
-rw-r--r-- 1 kafka kafka 2697871 Mar 26 07:11 00000000006877054274.log
-rw-r--r-- 1 kafka kafka 3792861 Mar 26 07:11 00000000006877066658.log
-rw-r--r-- 1 kafka kafka 931064 Mar 26 07:11 00000000006877084831.log
-rw-r--r-- 1 kafka kafka 118282 Mar 26 07:11 00000000006877089209.log
-rw-r--r-- 1 kafka kafka 1807647 Mar 26 07:11 00000000006877089458.log
-rw-r--r-- 1 kafka kafka 239104 Mar 26 07:11 00000000006877096136.log
-rw-r--r-- 1 kafka kafka 1835988 Mar 26 07:11 00000000006877097398.log
-rw-r--r-- 1 kafka kafka 575040 Mar 26 07:11 00000000006877104257.log
-rw-r--r-- 1 kafka kafka 390255 Mar 26 07:11 00000000006877106774.log
-rw-r--r-- 1 kafka kafka 3105807 Mar 26 07:11 00000000006877108341.log
-rw-r--r-- 1 kafka kafka 2086353 Mar 26 07:11 00000000006877120354.log
-rw-r--r-- 1 kafka kafka 2434773 Mar 26 07:12 00000000006877128121.log
-rw-r--r-- 1 kafka kafka 1432500 Mar 26 07:12 00000000006877137370.log
-rw-r--r-- 1 kafka kafka 1901742 Mar 26 07:12 00000000006877142384.log
-rw-r--r-- 1 kafka kafka 2310319 Mar 26 07:12 00000000006877149921.log
-rw-r--r-- 1 kafka kafka 2084367 Mar 26 07:12 00000000006877157735.log
-rw-r--r-- 1 kafka kafka 23317 Mar 26 07:12 00000000006877165836.log
-rw-r--r-- 1 kafka kafka 3715029 Mar 26 07:12 00000000006877165946.log
-rw-r--r-- 1 kafka kafka 2217586 Mar 26 07:12 00000000006877179092.log
-rw-r--r-- 1 kafka kafka 1133491 Mar 26 07:12 00000000006877187739.log
-rw-r--r-- 1 kafka kafka 1351694 Mar 26 07:12 00000000006877191615.log
-rw-r--r-- 1 kafka kafka 1397573 Mar 26 19:11 00000000006877195811.log
-rw-r--r-- 1 kafka kafka 1439494 Mar 28 00:37 00000000006877201824.log
-rw-r--r-- 1 kafka kafka 1679347 Mar 29 07:10 00000000006877207157.log
-rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000000000000000.timeindex
-rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000000000000000.index
-rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877089458.timeindex
-rw-r--r-- 1 kafka kafka 8 Mar 29 09:35 00000000006877089458.index
-rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877089209.timeindex
-rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000006877089209.index
-rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877084831.timeindex
-rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000006877084831.index
-rw-r--r-- 1 kafka kafka 36 Mar 29 09:35 00000000006877066658.timeindex
-rw-r--r-- 1 kafka kafka 24 Mar 29 09:35 00000000006877066658.index
..... 
.....
-rw-r--r-- 1 kafka kafka 10485760 Jun 24 14:52 00000000006877558794.index
-rw-r--r-- 1 kafka kafka 1235857 Jun 24 14:52 00000000006877558794.log

As you can see, the oldest log file has a 00000..00000 file name. 

Is there any version that we can upgrade to that will solve this issue for us? 

 

Thanks