[#HADOOP-3177] Expose DFSOutputStream.fsync API though the FileSystem interface

Hadoop Common

Expose DFSOutputStream.fsync API though the FileSystem interface

Created: 04/Apr/08 08:47 PM Updated: 08/Jul/09 04:43 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

0.18.0

Time Tracking:

Not Specified

File Attachments:

	File Name	Date Attached	Attached By	Size
	3177_20080603.patch	2008-06-03 08:43 PM	Tsz Wo (Nicholas), SZE	9 kB

Issue Links:

Blocker

This issue blocks:

~~HADOOP-1700~~ Append to files in HDFS

Hadoop Flags:	Reviewed
Release Note:	« Hide Added a new public interface Syncable which declares the sync() operation. FSDataOutputStream implements Syncable. If the wrappedStream in FSDataOutputStream is Syncalbe, calling FSDataOutputStream.sync() is equivalent to call wrappedStream.sync(). Otherwise, FSDataOutputStream.sync() is a no-op. Both DistributedFileSystem and LocalFileSystem support the sync() operation. Show » Added a new public interface Syncable which declares the sync() operation. FSDataOutputStream implements Syncable. If the wrappedStream in FSDataOutputStream is Syncalbe, calling FSDataOutputStream.sync() is equivalent to call wrappedStream.sync(). Otherwise, FSDataOutputStream.sync() is a no-op. Both DistributedFileSystem and LocalFileSystem support the sync() operation.
Resolution Date:	04/Jun/08 05:46 PM

Description

« Hide

In the current code, there is a DFSOutputStream.fsync() API that allows a client to flush all buffered data to the datanodes and also persist block locations on the namenode. This API should be exposed through the generic API in the org.hadoop.fs.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 28/Apr/08 05:37 PM

Do we really need to add this method to the FileSystem API? fsync seems DFS specific.

[ Permlink | « Hide ]

dhruba borthakur added a comment - 28/Apr/08 05:49 PM

I think it is sufficient (for now) for an application to cast a OutputStream to a DFSOutputStream and then invoke fsync on it. However, it is better if we expose this API on the generic FileSystem API so that an application can work same on all FileSystems.

[ Permlink | « Hide ]

Doug Cutting added a comment - 28/Apr/08 06:20 PM

I think sync is a fairly generic FileSystem operation.

http://java.sun.com/javase/6/docs/api/java/io/FileDescriptor.html#sync()

[ Permlink | « Hide ]

dhruba borthakur added a comment - 02/Jun/08 07:47 AM - edited

This issue blocks h-1700 because DFSOutputStream.fsync() is not a public API yet. More work is needed to make it accessible from an application.

One option would be to introduce FSDataOutputStream.fsync() API. It would invoke reflection to see if the wrapperStream as has a method named "fsync". If so, then it will invoke wrapperStream.fsync(), otherwise it will invoke wrapperStream.flush(). Does this sound reasonable?

[ Permlink | « Hide ]

Doug Cutting added a comment - 02/Jun/08 06:15 PM

> It would invoke reflection to see if the wrapperStream as has a method named "fsync".

Yuk. What is the reason not to add a sync() method to FileSystem and/or FSDataOutputStream?

[ Permlink | « Hide ]

dhruba borthakur added a comment - 02/Jun/08 11:07 PM

I am saying the same thing as you: let's add a new method FSDataOutputStream.fsync().

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 02/Jun/08 11:21 PM

The class for the wrapperStream is OutputStream. I think it should be java.io.FileOutputStream since we are doing FileSystem. Then, the new method FSDataOutputStream.fsync() just has to call wrapperStream.getFD().sync(). This will work for all FileSystem

For DFS, we need to define a new class, say DfsFileDescriptor, extending java.io.FileDescriptor and make DfsFileDescriptor.sync() calls DFSOutputStream.fsync().

For other FileSystem subclass, if getFD() is not defined, we could throw IOException("not supported").

[ Permlink | « Hide ]

Tom White added a comment - 03/Jun/08 01:25 PM

I think it should be java.io.FileOutputStream since we are doing FileSystem.

But FileOutputStream is tied to Java's File abstraction which isn't general enough for Hadoop FileSystems. Furthermore FileOutputStream#getFD is final, as is FileDescriptor, so we can't use it here.

How about an interface:

public interface Syncable {
  void sync() throws IOException;
}

(Or should it be "Synchable"?) Then make DFSOutputStream implement Syncable, so FSDataOutputStream - which is also a Syncable - can see if it can call sync() on the underlying stream.

What are the semantics of sync()? I think the expectation is that after sync returns the system has successfully sync'ed buffers to disk. So if this is not true, sync() should throw an exception. This is what java.io.FileDescriptor does. Using a subclass of IOException (java.io.SyncFailedException?) would make this easier for callers. I realize that this description is at odds with the current contract for DFSOutputStream#fsync, which doesn't guarantee that the data has been flushed to persistent storage, but I wondered whether DFSOutputStream could be strengthened to make this guarantee?

If the FileSystem doesn't support sync then do we get an exception when calling sync(), or is it a no op?

[ Permlink | « Hide ]

dhruba borthakur added a comment - 03/Jun/08 05:11 PM

The semantics of FileSystem.fsync is that the system makes every effort to put data on persistent storage. In the case of local file system, it will invoke the sync on the local file system. In the case of DFS, it will ensure that data has been flushed to OS buffers on all datanode(s) in the pipeline.

If a file-system does not support fsync, then it should be a no-op rather than an exception.This is similar to other calls like setReplication which returns success on local file systems even though there isn't any replication for local file system.