[TEZ-4088] Create in-memory ifile writer for transferring smaller payloads (follow up of TEZ-4075) - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.10.0
Component/s: None
Labels:
None

Description

~~TEZ-4075~~ enabled data transfer over DME for smaller payloads. This helps in reducing shuffle.

However, it still incurs disk IO cost (+flush) in producer side. It would be good to retain smaller payloads in mem, so that disk IO costs can be saved.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

TEZ-4088.1.patch
07/Oct/19 10:22
25 kB
Rajesh Balamohan
TEZ-4088.2.patch
07/Oct/19 16:34
25 kB
Rajesh Balamohan
TEZ-4088.3.patch
08/Oct/19 00:04
25 kB
Rajesh Balamohan
TEZ-4088.5.patch
10/Oct/19 22:21
26 kB
Rajesh Balamohan
TEZ-4088.6.patch
11/Oct/19 03:04
26 kB
Rajesh Balamohan

Issue Links

links to

GitHub Pull Request #50

Activity

People

Assignee:: Rajesh Balamohan

Reporter:: Rajesh Balamohan

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 07/Oct/19 07:03

Updated:: 25/Aug/20 15:00

Resolved:: 11/Oct/19 04:29

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

2h 20m