[#HADOOP-3250] Extend FileSystem API to allow appending to files

Hadoop Common

Extend FileSystem API to allow appending to files

Created: 14/Apr/08 07:57 PM Updated: 06/Oct/08 10:09 PM

Return to search

Component/s:

Affects Version/s:

None

Fix Version/s:

0.18.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
3250_20080529.patch	2008-05-30 12:02 AM	Tsz Wo (Nicholas), SZE	7 kB
3250_20080530.patch	2008-05-30 09:07 PM	Tsz Wo (Nicholas), SZE	11 kB

Issue Links:

Blocker

This issue blocks:

~~HADOOP-1700~~ Append to files in HDFS

Hadoop Flags:	Reviewed
Resolution Date:	02/Jun/08 07:52 PM

Description

« Hide

Provide an API to allow applications to append data to pre-existing files.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

dhruba borthakur added a comment - 14/Apr/08 08:05 PM

One option for this API would be a new method in the FileSystem.open() that takes in the pathname to be opened (and permissions, etc) and returns an FSDataOutputStream. The next write into this stream will occur at the end of the file.

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 21/Apr/08 08:59 PM

FileSystem.open currently returns a FSDataInputStream. Clearly, we cannot use an InputStream to do append. I guess we might want to define a new method, say "append", in the FileSystem API.

[ Permlink | « Hide ]

dhruba borthakur added a comment - 21/Apr/08 09:04 PM

My mistake. I wanted to that we should create another method called FileSystem.create() with an "append" flag.

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 21/Apr/08 09:16 PM

Literally, the word "create" does not mean or include append operation.

It seems to me that permission, overwrite, replication and blockSize do not make sense in append. This is the current create header.

public abstract FSDataOutputStream create(Path f,
      FsPermission permission,
      boolean overwrite,
      int bufferSize,
      short replication,
      long blockSize,
      Progressable progress) throws IOException;

[ Permlink | « Hide ]

Doug Cutting added a comment - 21/Apr/08 09:29 PM

I like the name 'append' for this.

public FSDataOutputStream append(Path f) {
  return append(f, getDefaultBufferSize(), null);
}
public FSDataOutputStream append(Path f, int bufferSize) {
  return append(f, buffersize, null);
}
public abstract FSDataOutputStream append(Path f, int bufferSize, Progressible progress);

[ Permlink | « Hide ]

dhruba borthakur added a comment - 21/Apr/08 11:33 PM

+1 on Doug's proposal.

If the path exists and is not being written to, then append() will return a OutputStream to the file. If the path does not exist, then append will generate a IOException.

The DFSClient will issue ClientProtocol.append() call to the namenode. If the file was being written to by another writer but it has exceeded the soft-lease period, then the namenode would start a distributed lease recovery. However, it will not make the client wait for the recovery to complete. The namenode will return an AlreadyBeingCreatedException as if the soft-limit has not yet expired.