[CARBONDATA-4042] Insert into select and CTAS launches fewer tasks(task count limited to number of nodes in cluster) even when target table is of no_sort

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: data-load, spark-integration
• Assignee: Unassigned
• Reporter: Venugopal Reddy K
• Created: 23/Oct/20 06:30
• Updated: 30/Oct/20 06:55
• Resolved: 29/Oct/20 11:23

Description

Issue:

At present, When we do insert into table select from or create table as select from, we lauch one single task per node. Whereas when we do a simple select * from table query, tasks launched are equal to number of carbondata files(CARBON_TASK_DISTRIBUTION default is CARBON_TASK_DISTRIBUTION_BLOCK). 

Thus, slows down the load performance of insert into select and ctas cases.

Refer Community discussion regd. task lauch

 

Suggestion:

Launch the same number of tasks as in select query for insert into select and ctas cases when the target table is of no-sort.