[HADOOP-14407] DistCp - Introduce a configurable copy buffer size - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.9.0
Fix Version/s: 2.9.0, 3.0.0-alpha4
Component/s: tools/distcp
Labels:
None

Hadoop Flags:

Reviewed
Release Note:
The copy buffer size can be configured via the new parameter <copybuffersize>. By default the <copybuffersize> is set to 8KB.

Description

Currently, the RetriableFileCopyCommand has a fixed copy buffer size of just 8KB. We have noticed in our performance tests that with bigger buffer sizes we saw upto ~3x performance boost. Hence, making the copy buffer size a configurable setting via the new parameter <copybuffersize>.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HADOOP-14407.001.patch
12/May/17 10:51
11 kB
Omkar Aradhya K S
HADOOP-14407.002.patch
18/May/17 02:00
11 kB
Yongjun Zhang
HADOOP-14407.002.patch
17/May/17 09:22
11 kB
Omkar Aradhya K S
HADOOP-14407.003.patch
18/May/17 07:15
12 kB
Omkar Aradhya K S
HADOOP-14407.004.branch2.patch
19/May/17 18:02
12 kB
Omkar Aradhya K S
HADOOP-14407.004.patch
18/May/17 17:24
12 kB
Omkar Aradhya K S
HADOOP-14407.004.patch
18/May/17 10:42
12 kB
Omkar Aradhya K S
HADOOP-14407.branch2.002.patch
25/May/17 02:16
13 kB
Yongjun Zhang
TotalTime-vs-CopyBufferSize.jpg
17/May/17 16:41
40 kB
Omkar Aradhya K S

Activity

People

Assignee:: Omkar Aradhya K S

Reporter:: Omkar Aradhya K S

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 10/May/17 10:16

Updated:: 24/Apr/18 20:50

Resolved:: 25/May/17 15:40