This is a bug in ChecksumFileSystem: if you call flush, it should flush the checksum stream too. But we perhaps don't have to fix that bug in this issue.
Changing flush() to throw an exception on all but HDFS (as proposed above) would not be good. This issue should improve flush() for HDFS, and not break it for all other filesystems.
Filing a separate issue to improve flush for ChecksumFileSystem would be good. This could either be done as I suggested above, by having FSOutputStream implement Seekable, but only implementing seek() in the local filesystem. Or instead, we could leave the FSOutputStream API alone, and ChecksumFileSystem could, when more output is written after a flush, throw an exception if the underlying FSOutputStream implementation doesn't implement Seekable. In either case, RawLocalFileSystem would implement Seekable for its FSOutputStream implementation, and ChecksumFileSystem could use this to rewind checksum output when data is appended after a flush().