Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3296

Support incremental dataload to datamap and other mv datamap enhancements

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently in carbondata we have datamaps like preaggregate, lucene, bloom, 
      mv and we have 
      lazy and non-lazy methods to load data to datamaps. But lazy load is not 
      allowed for datamaps 
      like preagg, lucene, bloom.but, it is allowed for mv datamap. In lazy load 
      of mv datamap, for 
      every rebuild(load to datamap) we load the complete data of main table and 
      overwrite the existing 
      segment in datamap based on datamap query. 

      This is very costly in terms of performance and we also need to support the 
      lazy and non-lazy load 
      for all the datamaps. This can help in reduce the actual dataload time to 
      main table and whenever 
      user wants, he can do the lazy load for the datamaps present for that table. 

      Basically we need not overwrite the existing data every time we load to 
      datamap, we need to increment 
      the data in new segments similar to main table. This will help to get 
      better performance. 

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              akashrn5 Akash R Nilugal
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 76.5h
                76.5h