[HIVE-7832] Do ORC dictionary check at a finer level and preserve encoding across stripes - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 0.14.0
Fix Version/s: 0.14.0
Component/s: None
Labels:
None

Release Note:

Hide
Added the new configuration to https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-ORCFileFormat

Show
Added the new configuration to https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-ORCFileFormat

Description

Currently ORC dictionary check happens while writing the stripe. Just before writing stripe if ratio of dictionary entries to total non-null rows is greater than threshold then the dictionary is discarded. Also, the decision of using dictionary or not is preserved across stripes. This sometimes leads to costly insertion cost of O(logn) for each stripes when there are too many distinct keys.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HIVE-7832.1.patch
21/Aug/14 19:10
12 kB
Prasanth Jayachandran
HIVE-7832.2.patch
22/Aug/14 07:27
12 kB
Prasanth Jayachandran
HIVE-7832.3.patch
22/Aug/14 18:23
13 kB
Prasanth Jayachandran
HIVE-7832.4.patch
26/Aug/14 07:42
25 kB
Prasanth Jayachandran
HIVE-7832.5.patch
26/Aug/14 18:51
24 kB
Prasanth Jayachandran
HIVE-7832.6.patch
26/Aug/14 19:01
23 kB
Prasanth Jayachandran
HIVE-7832.7.patch
27/Aug/14 17:17
23 kB
Prasanth Jayachandran
HIVE-7832.8.patch
30/Aug/14 05:57
23 kB
Prasanth Jayachandran
HIVE-7832.9.patch
02/Sep/14 07:42
23 kB
Prasanth Jayachandran

Issue Links

links to

Review Board

Activity

People

Assignee:: Prasanth Jayachandran

Reporter:: Prasanth Jayachandran

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 21/Aug/14 19:07

Updated:: 13/Nov/14 19:39

Resolved:: 02/Sep/14 20:53