[CARBONDATA-3296] Support incremental dataload to datamap and other mv datamap enhancements

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Akash R Nilugal
• Created: 18/Feb/19 13:59
• Updated: 06/May/19 08:38

Description

Currently in carbondata we have datamaps like preaggregate, lucene, bloom, 
mv and we have 
lazy and non-lazy methods to load data to datamaps. But lazy load is not 
allowed for datamaps 
like preagg, lucene, bloom.but, it is allowed for mv datamap. In lazy load 
of mv datamap, for 
every rebuild(load to datamap) we load the complete data of main table and 
overwrite the existing 
segment in datamap based on datamap query. 

This is very costly in terms of performance and we also need to support the 
lazy and non-lazy load 
for all the datamaps. This can help in reduce the actual dataload time to 
main table and whenever 
user wants, he can do the lazy load for the datamaps present for that table. 

Basically we need not overwrite the existing data every time we load to 
datamap, we need to increment 
the data in new segments similar to main table. This will help to get 
better performance. 

Sub-Tasks

• Incremental dat load support to datamap on single table: Resolved
• Incremental data load support for datamap on multiple tables: Open
• Support TableProperties from single parent table and restrict alter/delete/partition on mv: Resolved
• Support Partition with MV datamap & Show DataMap Status: Resolved
• Block complex data types and validate dmproperties in mv: Resolved
• Fix Concurrent dataloading Issue with mv: Resolved
• Add Documentation for MV datamap: Resolved