[HIVE-15676] Remove Bloom Filters from semi join reduction if it is too big.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Deepak Jaiswal
• Reporter: Deepak Jaiswal
• Created: 20/Jan/17 07:53
• Updated: 20/Jan/17 07:53

Description

Bloom filters themselves could become really big if the row count is high. Aggregating such bloom filters in reducers could be even more expensive. For e.g., a bloom filter for 100M rows can be as big as 170MB. Aggregating 100 such filters in reducer could end up taking 17GB of memory.