[HAMA-531] Data re-partitioning in BSPJobClient - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.6.1
Component/s: None
Labels:
None

Description

The re-partitioning the data is a very expensive operation. By the way, currently, we processes read/write operations sequentially using HDFS api in BSPJobClient from client-side. This causes potential too many open files error, contains HDFS overheads, and shows slow performance.

We have to find another way to re-partitioning data.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

patch_v04.txt
11/Dec/12 08:33
43 kB
Edward J. Yoon
patch_v03.txt
11/Dec/12 06:03
42 kB
Edward J. Yoon
patch_v02.txt
10/Dec/12 07:53
30 kB
Edward J. Yoon
patch.txt
09/Dec/12 07:19
19 kB
Edward J. Yoon
HAMA-531_final.patch
21/May/12 18:41
41 kB
Thomas Jungblut
HAMA-531_2.patch
21/May/12 18:13
32 kB
Thomas Jungblut
HAMA-531_1.patch
21/May/12 14:12
19 kB
Thomas Jungblut

Issue Links

is related to

HAMA-561 Support consuming partitioned files

Open

HAMA-577 Partitioned data should be optionally written to disk

Resolved

relates to

HAMA-700 BSPPartitioner should be configurable to be optional and allow input format conversion

Resolved

Activity

People

Assignee:: Edward J. Yoon

Reporter:: Edward J. Yoon

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 09/Mar/12 05:44

Updated:: 24/Dec/12 00:10

Resolved:: 12/Dec/12 05:38