[HADOOP-15887] Add an option to avoid writing data locally in Distcp - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Patch Available
Priority: Major
Resolution: Unresolved
Affects Version/s: 2.8.2, 3.0.0
Fix Version/s: None
Component/s: tools/distcp
Labels:
None

Target Version/s:

3.5.0

Description

When copying large amount of data from one cluster to another via Distcp, and the Distcp jobs run in the target cluster, the datanode local usage would be imbalanced. Because the default placement policy chooses the local node to store the first replication.

In https://issues.apache.org/jira/browse/HDFS-3702 we add a flag in DFSClient to avoid replicating to the local datanode. We can make use of this flag in Distcp.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HADOOP-15887.005.patch
02/Jan/20 01:13
18 kB
Tao Jie
HADOOP-15887.004.patch
31/Dec/19 13:26
17 kB
Tao Jie
HADOOP-15887.003.patch
31/Dec/19 10:34
17 kB
Tao Jie
HADOOP-15887.002.patch
02/Nov/18 06:26
18 kB
Tao Jie
HADOOP-15887.001.patch
30/Oct/18 05:57
11 kB
Tao Jie

Issue Links

links to

GitHub Pull Request #1794

Activity

People

Assignee:: Tao Jie

Reporter:: Tao Jie

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 30/Oct/18 05:55

Updated:: 04/Jan/24 12:34