Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.
A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.
Attachments
Attachments
Issue Links
- blocks
-
HADOOP-1700 Append to files in HDFS
- Closed
- depends upon
-
HADOOP-1927 a datanode in a write pipeline should report an error if the next datanode in the pipeline reported an error
- Closed
- incorporates
-
HDFS-67 /tmp/hadoop-${user}/dfs/tmp/tmp/client-${long}.tmp is not cleanup correctly
- Resolved
- is related to
-
HADOOP-445 Parallel data/socket writing for DFSOutputStream
- Closed
Activity
Link |
This issue depends upon |
Link |
This issue depends on |
Component/s | dfs [ 12310710 ] |
Status | Resolved [ 5 ] | Closed [ 6 ] |
Resolution | Fixed [ 1 ] | |
Status | Patch Available [ 10002 ] | Resolved [ 5 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Attachment | clientDiskBuffer27.patch [ 12373344 ] |
Attachment | clientDiskBuffer27.patch [ 12373337 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Attachment | clientDiskBuffer26.patch [ 12373299 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Link |
This issue incorporates |
Attachment | clientDiskBuffer25.patch [ 12373225 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Attachment | clientDiskBuffer24.patch [ 12373189 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Attachment | clientDiskBuffer24.patch [ 12373147 ] |
Attachment | clientDiskBuffer23.patch [ 12373138 ] |
Attachment | clientDiskBuffer23.patch [ 12373133 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Attachment | clientDiskBuffer22.patch [ 12373117 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Attachment | clientDiskBuffer21.patch [ 12373081 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Attachment | clientDiskBuffer20.patch [ 12373048 ] |
Status | Patch Available [ 10002 ] | Open [ 1 ] |
Status | Open [ 1 ] | Patch Available [ 10002 ] |
Attachment | clientDiskBuffer19.patch [ 12372949 ] |
Attachment | clientDiskBuffer18.patch [ 12372728 ] |
Attachment | clientDiskBuffer17.patch [ 12372646 ] |
Attachment | clientDiskBuffer16.patch [ 12372326 ] |
Link |
This issue is related to |
Attachment | clientDiskBuffer15.patch [ 12371931 ] |
Attachment | clientDiskBuffer14.patch [ 12371568 ] |
Attachment | clientDiskBuffer12.patch [ 12371209 ] |
Attachment | clientDiskBuffer11.patch [ 12371080 ] |
Attachment | clientDiskBuffer10.patch [ 12371005 ] |
Attachment | clientDiskBuffer9.patch [ 12370884 ] |
Attachment | clientDiskBuffer8.patch [ 12370340 ] |
Attachment | clientDiskBuffer7.patch [ 12369962 ] |
Attachment | DataTransferProtocol.html [ 12369626 ] |
Attachment | DataTransferProtocol.doc [ 12369625 ] |
Attachment | clientDiskBuffer6.patch [ 12369417 ] |
Attachment | clientDiskBuffer2.patch [ 12368891 ] |
Attachment | clientDiskBuffer.patch [ 12368789 ] |
Link |
This issue blocks |
Link |
This issue depends on |
Fix Version/s | 0.16.0 [ 12312740 ] | |
Summary | DFS client can allow user to write data to the next block while uploading previous block to HDFS | Remove the DFS Client disk-based cache |
Issue Type | Bug [ 1 ] | Improvement [ 4 ] |
Field | Original Value | New Value |
---|---|---|
Assignee | dhruba borthakur [ dhruba ] |