Kafka does not call fsync() on directory when new log segment is created and flushed to disk.
The problem is that following sequence of calls doesn't guarantee file durability:
fd = open("log", O_RDWR | O_CREATE); // suppose open creates "log"
If system crashes after fsync() but before parent directory have been flushed to disk, the log file can disappear.
This is true at least for ext4 on Linux.
Proposed solution is to flush directory when flush() is called for the first time.