[KAFKA-4127] Possible data loss

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Environment: Normal three node Kafka cluster. All machines running linux.
• Assignee: Unassigned
• Reporter: Ramnatthan Alagappan
• Created: 05/Sep/16 18:56
• Updated: 06/Sep/16 17:42
• Resolved: 06/Sep/16 17:41

Description

I am running a three node Kakfa cluster. ZooKeeper runs in a standalone mode.

When I create a new message topic, I see the following sequence of system calls:

mkdir("/appdir/my-topic1-0")
creat("/appdir/my-topic1-0/00000000000000000000.log")

I have configured Kafka to write the messages persistently to the disk before acknowledging the client. Specifically, I have set flush.interval_messages to 1, min_insync_replicas to 3, and disabled dirty election. Now, I insert a new message into the created topic.

I see that Kafka writes the message to the log file and flushes the data down to disk by carefully fsync'ing the log file. I get an acknowledgment back from the cluster after the message is safely persisted on all three replicas and written to disk.

Unfortunately, Kafka can still lose data since it does not explicitly fsync the directory to persist the directory entries of the topic directory and the log file. If a crash happens after acknowledging the client, it is possible for Kafka lose the directory entry for the topic directory or the log file. Many systems carefully issue fsync to the parent directory when a new file or directory is created. This is required for the file to be completely persisted to the disk.

Issue Links

• duplicates: KAFKA-3968 fsync() is not called on parent directory when new FileMessageSet is flushed to disk