[HCATALOG-328] HCatLoader should report its input size so pig can estimate the number of reducers - ASF JIRA

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.5
Component/s: None
Labels:
None

Description

~~PIG-2573~~ updated the way pig estimates the number of reducers, adding the ability to ask loaders for the input size. HCatLoader needs to report the size of its input so pig can estimate the number of reducers. Currently, since the loader does not report its size, and "db.table" does not match the whitelisted filesystems for pig to look up size itself, size is reported as 0 and 1 reducer is used.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HCATALOG-328_2.patch
22/Mar/12 23:43
59 kB
Travis Crawford
HCATALOG-328_3.patch
23/Mar/12 02:55
62 kB
Travis Crawford
HCATALOG-328_4.patch
23/Mar/12 22:47
7 kB
Travis Crawford
HCATALOG-328.patch
22/Mar/12 22:59
51 kB
Travis Crawford

Activity

People

Assignee:: Travis Crawford

Reporter:: Travis Crawford

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 16/Mar/12 22:31

Updated:: 26/Mar/12 17:01

Resolved:: 26/Mar/12 16:54