HDFS sink calls hflush() (or sync()) on the FSDataOutputStream which will flush client buffers, but will not update the output file size on the NameNode (see
HDFS-5478) while it is being written, only after it is closed.
It would be nice to allow users to trigger updating the file length (which also syncs file data to disk, see
This could be done via new hdfs.hsyncInterval, hdfs.hsyncSize and hdfs.hsyncCount configuration options.
A workaround is to roll the output file more often, but that leads to many small files which may be worse than putting extra load on the NameNode by calling hsync(...) multiple times during write, right?