Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7322

Fix race condition between log cleaner thread and log retention thread when topic cleanup policy is updated

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: log
    • Labels:
      None

      Description

      The deletion thread will grab the log.lock when it tries to rename log segment and schedule for actual deletion.

      The compaction thread only grabs the log.lock when it tries to replace the original segments with the cleaned segment. The compaction thread doesn't grab the log when it reads records from the original segments to build offsetmap and new segments. As a result, if both deletion and compaction threads work on the same log partition. We have a race condition. 

      This race happens when the topic cleanup policy is updated on the fly.  

      One case to hit this race condition:

      1: topic clean up policy is "compact" initially 

      2: log cleaner (compaction) thread picks up the partition for compaction and still in progress

      3: the topic clean up policy has been updated to "deletion"

      4: retention thread pick up the topic partition and delete some old segments.

      5: log cleaner thread reads from the deleted log and raise an IO exception. 

       

      The proposed solution is to use "inprogress" map that cleaner manager has to protect such a race.

       

        Attachments

          Activity

            People

            • Assignee:
              xiongqiwu xiongqi wu
              Reporter:
              xiongqiwu xiongqi wu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: