Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9835

Race condition with concurrent write allows reads above high watermark

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0, 2.3.0, 2.4.0
    • Fix Version/s: 2.4.2, 2.5.1
    • Component/s: None
    • Labels:


      Kafka's log implementation serializes all writes using a lock, but allows multiple concurrent reads while that lock is held. The `FileRecords` class contains the core implementation. Reads to the log create logical slices of `FileRecords` which are then passed to the network layer for sending. An abridged version of the implementation of `slice` is provided below:

          public FileRecords slice(int position, int size) throws IOException {
              int end = this.start + position + size;
              // handle integer overflow or if end is beyond the end of the file
              if (end < 0 || end >= start + sizeInBytes())
                  end = start + sizeInBytes();
              return new FileRecords(file, channel, this.start + position, end, true);

      The `size` parameter here is typically derived from the fetch size, but is upper-bounded with respect to the high watermark. The two calls to `sizeInBytes` here are problematic because the size of the file may change in between them. Specifically a concurrent write may increase the size of the file after the first call to `sizeInBytes` but before the second one. In the worst case, when `size` defines the limit of the high watermark, this can lead to a slice containing uncommitted data.




            • Assignee:
              hachikuji Jason Gustafson
              hachikuji Jason Gustafson


              • Created:

                Issue deployment