[KAFKA-16424] truncated logs will be left undeleted after alter dir completion

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.7.0
• Fix Version/s: 3.8.0
• Assignee: PoAn Yang
• Reporter: Luke Chen
• Created: 26/Mar/24 09:56
• Updated: 10/Sep/24 09:05
• Resolved: 24/Apr/24 09:51

Description

When doing log dir movement, we'll create a temp future replica with the dir named: topic-partition.uniqueId-future, ex: t3-0.9af8e054dbe249cf9379a210ec449af8-future. After the log dir movement completed, we'll rename the future log dir to the normal log dir, in the above case, it'll be "t3" only.

So, if there are some logs to be deleted during the log dir movement, we'll send for a scheduler to do the deletion later (here). However, when the log dir movement completed, the future log is renamed, the async log deletion will fail with no file existed error:

 

[2024-03-26 17:35:10,809] INFO [LocalLog partition=t3-0, dir=/tmp/kraft-broker-logs] Deleting segment files LogSegment(baseOffset=0, size=0, lastModifiedTime=0, largestRecordTimestamp=-1) (kafka.log.LocalLog$)
[2024-03-26 17:35:10,810] INFO Failed to delete log /tmp/kraft-broker-logs/t3-0.9af8e054dbe249cf9379a210ec449af8-future/00000000000000000000.log.deleted because it does not exist. (org.apache.kafka.storage.internals.log.LogSegment)
[2024-03-26 17:35:10,811] INFO Failed to delete offset index /tmp/kraft-broker-logs/t3-0.9af8e054dbe249cf9379a210ec449af8-future/00000000000000000000.index.deleted because it does not exist. (org.apache.kafka.storage.internals.log.LogSegment)
[2024-03-26 17:35:10,811] INFO Failed to delete time index /tmp/kraft-broker-logs/t3-0.9af8e054dbe249cf9379a210ec449af8-future/00000000000000000000.timeindex.deleted because it does not exist. (org.apache.kafka.storage.internals.log.LogSegment) 

I think we could consider fall back to the normal log dir if the future log dir cannot find the files. That is, when the file cannot be found under "t3-0.9af8e054dbe249cf9379a210ec449af8-future" dir, then try to find "t3" folder, and delete the file. Because the file is already having the suffix with ".delete", it should be fine if we delete them.

Issue Links

• is related to: KAFKA-17508 Adding some guard for fallback deletion logic
• links to: GitHub Pull Request #15616