[CARBONDATA-3347] support SORT_COLUMNS modification

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: spark-integration
• Assignee: David Cai
• Reporter: David Cai
• Created: 09/Apr/19 09:26
• Updated: 07/May/20 02:06
• Resolved: 07/May/20 02:06

Description

Background

Now SORT_COLUMNS can’t be modified after the table is created. If we want to modify SORT_COLUMNS on this table, we need to create a new table and migrate data. If the data is huge, the migration will take a long time and even impact the user business.

SORT_SCOPE in table properties can be modified now. And we can specify new SORT_SCOPE during data loading. Carbon index file will mark whether this segment is sorted or not. So the different segments maybe have different SORT_SCOPE.

Motivation

After the table is created, the user can adjust SORT_SCOPE/SORT_COLUMNS according to their business. History segments will still use old SORT_SCOPE/SORT_COLUMNS, but the user also can resort old segments one by one if need.

But we still suggest the user give a proper SORT_SCOPE/SORT_COLUMNS when they create the table because the modification will take many resources to resort data of old segments.

 

please check design doc for more detail.

sort_columns modification_v2.pdf

Sub-Tasks

• support alter table property SORT_COLUMNS: Resolved
• add is_sorted and sort_columns information into show segments: Resolved
• enhance custom compaction to support resort single segment: Resolved