[HADOOP-13169] Randomize file list in SimpleCopyListing - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.8.0
Component/s: tools/distcp
Labels:
None

Target Version/s:

2.8.0
Hadoop Flags:

Reviewed

Description

When copying files to S3, based on file listing some mappers can get into S3 partition hotspots. This would be more visible, when data is copied from hive warehouse with lots of partitions (e.g date partitions). In such cases, some of the tasks would tend to be a lot more slower than others. It would be good to randomize the file paths which are written out in SimpleCopyListing to avoid this issue.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HADOOP-13169-branch-2-010.patch
17/Sep/16 00:38
15 kB
Rajesh Balamohan
HADOOP-13169-branch-2-009.patch
16/Sep/16 02:55
15 kB
Rajesh Balamohan
HADOOP-13169-branch-2-008.patch
14/Sep/16 16:03
14 kB
Rajesh Balamohan
HADOOP-13169-branch-2-007.patch
14/Sep/16 10:40
13 kB
Rajesh Balamohan
HADOOP-13169-branch-2-006.patch
14/Sep/16 09:31
14 kB
Rajesh Balamohan
HADOOP-13169-branch-2-005.patch
13/Sep/16 02:36
12 kB
Rajesh Balamohan
HADOOP-13169-branch-2-004.patch
12/Sep/16 04:41
8 kB
Rajesh Balamohan
HADOOP-13169-branch-2-003.patch
08/Sep/16 05:52
8 kB
Rajesh Balamohan
HADOOP-13169-branch-2-002.patch
18/May/16 06:43
6 kB
Rajesh Balamohan
HADOOP-13169-branch-2-001.patch
18/May/16 04:23
7 kB
Rajesh Balamohan

Issue Links

is depended upon by

HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance

Resolved

Activity

People

Assignee:: Rajesh Balamohan

Reporter:: Rajesh Balamohan

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 18/May/16 02:21

Updated:: 28/Feb/17 17:24

Resolved:: 19/Sep/16 22:28