[CARBONDATA-3879] Filtering Segmets Optimazation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: data-query
• Assignee: Unassigned
• Reporter: Xingjun Hao
• Created: 29/Jun/20 14:15
• Updated: 06/Aug/20 09:32
• Resolved: 06/Aug/20 09:32

Description

During filter segments flow, there are a lot of LIST.CONTAINS, which has heavy time overhead when there are tens of thousands segments.

For example, if there are 50000 segments. it will trigger LIST.CONTAINS  for each segment, the LIST also has about 50000 elements. so the time complexity will be O（50000 * 50000 ）