[TEZ-3605] Detect and prune empty partitions for the Ordered case - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.9.0
Component/s: None
Labels:
None

Hadoop Flags:

Reviewed

Description

Analogous to the Unordered case we should not have empty partition entries/segments in the Ordered/DefaultSorter case. This will save writing unnecessary data.
Additionally, with tez_shuffle feature (~~TEZ-3334~~), in a heavily auto reduced job, this change would allow not fetching empty partitions and then throwing them away.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

TEZ-3605.013.patch
27/Jun/17 19:58
29 kB
Kuhu Shukla
TEZ-3605.012.patch
26/Jun/17 21:07
29 kB
Kuhu Shukla
TEZ-3605.011.patch
26/Jun/17 18:29
29 kB
Kuhu Shukla
TEZ-3605.010.patch
30/May/17 21:02
13 kB
Kuhu Shukla
TEZ-3605.009.patch
22/May/17 19:52
13 kB
Kuhu Shukla
TEZ-3605.008.patch
18/May/17 17:40
10 kB
Kuhu Shukla
TEZ-3605.007.patch
18/May/17 16:15
10 kB
Kuhu Shukla
TEZ-3605.006.patch
15/Mar/17 20:44
21 kB
Kuhu Shukla
TEZ-3605.005.patch
07/Mar/17 21:49
20 kB
Kuhu Shukla
TEZ-3605.004.patch
07/Mar/17 14:39
19 kB
Kuhu Shukla
TEZ-3605.003.patch
06/Mar/17 19:07
19 kB
Kuhu Shukla
TEZ-3605.002.patch
06/Mar/17 14:58
19 kB
Kuhu Shukla
TEZ-3605.001.patch
04/Mar/17 00:12
19 kB
Kuhu Shukla

Issue Links

is duplicated by

TEZ-941 Avoid writing out empty partitions in Sorter implementations

Open

Activity

People

Assignee:: Kuhu Shukla

Reporter:: Kuhu Shukla

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 03/Feb/17 19:01

Updated:: 22/Aug/17 00:03

Resolved:: 28/Jun/17 17:52