[CARBONDATA-2433] Executor OOM because of GC when blocklet pruning is done using Lucene datamap

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Assignee: Manish Gupta
• Reporter: Manish Gupta
• Created: 03/May/18 15:34
• Updated: 22/May/18 10:59
• Resolved: 21/May/18 13:45

Description

While seraching using lucene it creates a PriorityQueue to hold the documents. As size is not specified by default the PriorityQueue size is equal to the number of lucene documents. As the docuemnts start getting added to the heap the GC time increases and after some time task fails due to excessive GC and executor OOM occurs.

Reference blog:

*http://lucene.472066.n3.nabble.com/Optimization-of-memory-usage-in-PriorityQueue-td590355.html*

Issue Links

• links to: GitHub Pull Request #2267
• links to: GitHub Pull Request #2269