Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3968

fsync() is not called on parent directory when new FileMessageSet is flushed to disk

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.9.0.1, 0.10.0.0
    • 3.0.0
    • log
    • Linux, ext4 filesystem
    • Patch

    Description

      Kafka does not call fsync() on directory when new log segment is created and flushed to disk.

      The problem is that following sequence of calls doesn't guarantee file durability:

      fd = open("log", O_RDWR | O_CREATE); // suppose open creates "log"
      write(fd);
      fsync(fd);

      If system crashes after fsync() but before parent directory have been flushed to disk, the log file can disappear.
      This is true at least for ext4 on Linux.

      Proposed solution is to flush directory when flush() is called for the first time.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tson Cong Ding
            nepal Andrey Neporada
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment