[CARBONDATA-4182] Performance issue when multiple load happeneing to same table for same interval with 2 MVs.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Component/s: core
• Environment: Apache carbon 2.1.0
• Assignee: Unassigned
• Reporter: suyash yadav
• Created: 12/May/21 04:52
• Updated: 12/May/21 04:52

Description

Hi Team,

We need your help to resolve one of the performance issue that we are facing. Please see below details about the table structure and schema implemented at our end:

1.We have 25 tables and 2 MVS created for these tables for hour and day granularity.
2.One table can have more than 1 can for same interval and whenver multiple csv are there as an input to these tables then sequential loading will take place.
3.for different tables data loading is parallal but whenver 2 csvs are there for same table then for that table sequential load will happen by one and then other csv respectivly.
4.We have observed for 1minute of csv to be loaded into the table it is taking approximalty 345 to 60 minutes which is creating a huge backlog.

We need your help to resolve this performanec issue as there will be no use of 15 minutes of data will take more than 15 minutes to load. Users are not going to wait and will get fed up by this slowness.

Kindly advice.

Regards
Suyash Yadav