[CARBONDATA-2853] Add min/max index for streaming segment

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Assignee: David Cai
• Reporter: David Cai
• Created: 13/Aug/18 08:37
• Updated: 11/Sep/18 02:59
• Resolved: 05/Sep/18 06:22

Description

Streaming index file in stream segment adds min/max meta index for each streaming file during streaming ingestion. So the filter query can use the min/max index to prune the streaming files to reduce the number of the spark tasks in the driver side. Streaming file adds min/max into the blocklet header, so the filter query can skip data during scanning file.

Issue Links

• links to: GitHub Pull Request #2644