[KAFKA-6347] Starting offset breach based log segment deletion never considers active segment

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: core
• Assignee: Unassigned
• Reporter: Guozhang Wang
• Created: 12/Dec/17 00:17
• Updated: 12/Dec/17 00:17

Description

This observation is related to KIP-107: in Log#deleteLogStartOffsetBreachedSegments, we designed the predicate to enforce that the current segment can only be considered for deletion, when it 1) has the next segment, 2) the next segment's starting offset is no larger than the log start offset.

def shouldDelete(segment: LogSegment, nextSegmentOpt: Option[LogSegment]) =
      nextSegmentOpt.exists(_.baseOffset <= logStartOffset)

This means that, the current active segment would never be considered for purging. Normally this is OK given that we will eventually roll out a new segment. However, with the default size of segment of 1GB it means that with the purge data API we are in the worst case not being able to purge the data for up to 1GB per topic partition.