|
If you were logged in you would be able to see more operations.
|
|
|
|
File Attachments:
|
|
|
Issue Links:
|
Blocker
|
|
|
|
This issue is blocked by:
|
|
HADOOP-2565
DFSPath cache of FileStatus can become stale
|
|
|
|
 |
HADOOP-3503
Race condition when client and namenode start block recovery simultaneously
|
|
|
|
|
|
HADOOP-3113
DFSOututStream.flush() should flush data to real block file on DataNode.
|
|
|
|
|
|
HADOOP-3176
Change lease record when a open-for-write-file gets renamed
|
|
|
|
|
|
HADOOP-2656
Support for upgrading existing cluster to facilitate appends to HDFS files
|
|
|
|
|
|
HADOOP-2655
Copy on write for data and metadata files in the presence of snapshots
|
|
|
|
|
|
HADOOP-3161
TestFileAppend fails on Mac since HADOOP-2655 was committed
|
|
|
|
|
|
HADOOP-2658
Design and Implement a Test Plan to support appends to HDFS files
|
|
|
|
|
|
HADOOP-3250
Extend FileSystem API to allow appending to files
|
|
|
|
|
|
HADOOP-3201
namenode should be able to retrieve block metadata from a datanode
|
|
|
|
|
|
HADOOP-3515
Protocol changes to allow appending to the last partial crc chunk of a file
|
|
|
|
|
|
HADOOP-3177
Expose DFSOutputStream.fsync API though the FileSystem interface
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Dependants
|
|
This issue depends on:
|
|
|
|
|
|
HADOOP-3283
Need a mechanism for data nodes to update generation stamps.
|
|
|
|
|
|
Reference
|
|
This issue relates to:
|
|
|
|
HADOOP-3834 Checkin the design document for HDFS appends into source control repository
|
|
|
|
 |
HADOOP-89
files are not visible until they are closed
|
|
|
|
|
|
HADOOP-3832
Create more unit tests for testing HDFS appends
|
|
|
|
|
|
HADOOP-3329
DatanodeDescriptor objects stored in FSImage may be out dated.
|
|
|
|
|
|
HADOOP-3241
DFSFileInfo should also have field to say if the file is underconstrction
|
|
|
|
|
|
HADOOP-1497
Possibility of duplicate blockids if dead-datanodes come back up after corresponding files were deleted
|
|
|
|
|
|
HADOOP-2657
Enhancements to DFSClient to support flushing data at any point in time
|
|
|
|
|
|
|
HADOOP-3833 Create a benchmark to measure performance of "append" to HDFS files
|
|
|
|
|
|
This issue is related to:
|
|
|
|
HDFS-200 In HDFS, sync() not yet guarantees data available to the new readers
|
|
|
|
|
|
|
|
|
|
dependent
|
|
|
|
This issue is depended upon by:
|
|
HADOOP-3790
Add more unit tests to test appending to files in HDFS
|
|
|
|
|
|
|
| Hadoop Flags: |
Reviewed, Incompatible change
|
| Release Note: |
Introduced append operation for HDFS files.
Introduced append operation for HDFS files.
|
| Resolution Date: |
25/Jul/08 06:09 PM
|
Request for being able to append to files in HDFS has been raised a couple of times on the list of late. For one example, see http://www.nabble.com/HDFS%2C-appending-writes-status-tf3848237.html#a10916193 . Other mail describes folks' workarounds because this feature is lacking: e.g. http://www.nabble.com/Loading-data-into-HDFS-tf4200003.html#a12039480 (Later on this thread, Jim Kellerman re-raises the HBase need of this feature). HADOOP-337 'DFS files should be appendable' makes mention of file append but it was opened early in the life of HDFS when the focus was more on implementing the basics rather than adding new features. Interest fizzled. Because HADOOP-337 is also a bit of a grab-bag – it includes truncation and being able to concurrently read/write – rather than try and breathe new life into HADOOP-337, instead, here is a new issue focused on file append. Ultimately, being able to do as the google GFS paper describes – having multiple concurrent clients making 'Atomic Record Append' to a single file would be sweet but at least for a first cut at this feature, IMO, a single client appending to a single HDFS file letting the application manage the access would be sufficent.
|
|
Description
|
Request for being able to append to files in HDFS has been raised a couple of times on the list of late. For one example, see http://www.nabble.com/HDFS%2C-appending-writes-status-tf3848237.html#a10916193. Other mail describes folks' workarounds because this feature is lacking: e.g. http://www.nabble.com/Loading-data-into-HDFS-tf4200003.html#a12039480 (Later on this thread, Jim Kellerman re-raises the HBase need of this feature). HADOOP-337 'DFS files should be appendable' makes mention of file append but it was opened early in the life of HDFS when the focus was more on implementing the basics rather than adding new features. Interest fizzled. Because HADOOP-337 is also a bit of a grab-bag – it includes truncation and being able to concurrently read/write – rather than try and breathe new life into HADOOP-337, instead, here is a new issue focused on file append. Ultimately, being able to do as the google GFS paper describes – having multiple concurrent clients making 'Atomic Record Append' to a single file would be sweet but at least for a first cut at this feature, IMO, a single client appending to a single HDFS file letting the application manage the access would be sufficent.
|
Show » |
made changes - 10/Aug/07 12:32 AM
| Field |
Original Value |
New Value |
|
Link
|
|
This issue relates to HADOOP-89
[ HADOOP-89
]
|
made changes - 19/Oct/07 09:52 AM
|
Attachment
|
|
Appends.htm
[ 12367999
]
|
made changes - 19/Oct/07 09:55 AM
|
Attachment
|
|
Appends.doc
[ 12368000
]
|
made changes - 19/Oct/07 04:34 PM
|
Attachment
|
|
Appends-1.xhtml
[ 12368024
]
|
made changes - 15/Nov/07 07:56 AM
|
Attachment
|
|
Appends.doc
[ 12369579
]
|
made changes - 15/Nov/07 07:56 AM
|
Attachment
|
Appends.doc
[ 12368000
]
|
|
made changes - 16/Nov/07 06:39 AM
|
Attachment
|
|
Appends.doc
[ 12369638
]
|
made changes - 16/Nov/07 06:40 AM
|
Attachment
|
|
Appends.html
[ 12369639
]
|
made changes - 16/Nov/07 06:41 AM
|
Attachment
|
Appends-1.xhtml
[ 12368024
]
|
|
made changes - 16/Nov/07 06:41 AM
|
Attachment
|
Appends.doc
[ 12369579
]
|
|
made changes - 16/Nov/07 06:41 AM
|
Attachment
|
Appends.htm
[ 12367999
]
|
|
made changes - 29/Nov/07 05:53 PM
|
Attachment
|
|
Appends.doc
[ 12370562
]
|
made changes - 26/Dec/07 02:49 AM
|
Attachment
|
|
append.patch
[ 12372198
]
|
made changes - 26/Dec/07 03:16 AM
|
Affects Version/s
|
|
0.15.1
[ 12312850
]
|
|
Status
|
Open
[ 1
]
|
Patch Available
[ 10002
]
|
made changes - 03/Feb/08 09:31 AM
|
Status
|
Patch Available
[ 10002
]
|
Open
[ 1
]
|
made changes - 07/May/08 10:53 PM
made changes - 29/Jun/08 10:49 AM
|
Attachment
|
|
append3.patch
[ 12384904
]
|
made changes - 29/Jun/08 10:50 AM
|
Assignee
|
|
dhruba borthakur
[ dhruba
]
|
made changes - 16/Jul/08 10:04 PM
|
Status
|
Open
[ 1
]
|
Patch Available
[ 10002
]
|
made changes - 18/Jul/08 05:14 PM
|
Status
|
Patch Available
[ 10002
]
|
Open
[ 1
]
|
made changes - 19/Jul/08 01:25 AM
|
Status
|
Open
[ 1
]
|
Patch Available
[ 10002
]
|
made changes - 23/Jul/08 06:43 PM
made changes - 24/Jul/08 01:24 PM
|
Status
|
Patch Available
[ 10002
]
|
Open
[ 1
]
|
made changes - 24/Jul/08 01:28 PM
|
Hadoop Flags
|
[Reviewed]
|
[Incompatible change, Reviewed]
|
|
Release Note
|
|
The protocol used by the DFSClient to Datanode and Namenode has been modified.
A new HadoopMetric named "FilesAppended" is added. This metric shows the number of times any file was opened for appends.
|
|
Status
|
Open
[ 1
]
|
Patch Available
[ 10002
]
|
made changes - 25/Jul/08 06:07 PM
|
Fix Version/s
|
|
0.19.0
[ 12313211
]
|
|
Hadoop Flags
|
[Reviewed, Incompatible change]
|
[Incompatible change, Reviewed]
|
made changes - 25/Jul/08 06:09 PM
|
Hadoop Flags
|
[Reviewed, Incompatible change]
|
[Incompatible change, Reviewed]
|
|
Resolution
|
|
Fixed
[ 1
]
|
|
Status
|
Patch Available
[ 10002
]
|
Resolved
[ 5
]
|
made changes - 09/Oct/08 01:03 AM
|
Link
|
|
This issue is related to HADOOP-4379
[ HADOOP-4379
]
|
made changes - 21/Oct/08 10:43 PM
|
Release Note
|
The protocol used by the DFSClient to Datanode and Namenode has been modified.
A new HadoopMetric named "FilesAppended" is added. This metric shows the number of times any file was opened for appends.
|
Introduced append operation for HDFS files.
|
|
Hadoop Flags
|
[Reviewed, Incompatible change]
|
[Incompatible change, Reviewed]
|
made changes - 20/Nov/08 11:38 PM
|
Status
|
Resolved
[ 5
]
|
Closed
[ 6
]
|
made changes - 08/Jul/09 04:42 PM
|
Component/s
|
dfs
[ 12310710
]
|
|
|
New Feature
