[CARBONDATA-3985] Optimize the segment-timestamp file clean up

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: core, spark-integration
• Assignee: Unassigned
• Reporter: suwen
• Created: 14/Sep/20 01:55
• Updated: 16/Sep/20 03:56

Description

For data update, in the CarbonProjectForUpdateCommand process, after the delete delta file is generated, the status of each segment is checked. If the status is not successful, all the segment directories are traversed to clean up the timestamp corresponding .carbondata, .carbonindex and .deletedelta files.

If a great many segments have been generated in the Partion directory, it will be very time-consuming.

In fact, in the process of cleaning up timestamp files, we only need to clean up the files in the Segment directory involved in this update.

In the process of generating delete delta, record the segment path involved in this update; after entering the checkAndUpdateStatusFiles() function, if a segment status is found to be not successful, it will be cleaned directly according to the segment path list that has been recorded during generating delete delta, without searching all the segment directories.