[CARBONDATA-4037] Improve the table status and segment file writing

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Assignee: Unassigned
• Reporter: SHREELEKHYA GAMPA
• Created: 19/Oct/20 16:01
• Updated: 22/Apr/21 09:56
• Resolved: 22/Apr/21 09:56

Description

Currently, we update table status and segment files multiple times for a single iud/merge/compact operation and delete the index files immediately after merge. When concurrent queries are run, there may be situations like user query is trying to access the segment index files and they are not present, which is availability issue.

• To solve above issue, we can make mergeindex files generation mandatory and fail load/compaction if mergeindex fails. Then if merge index is success, update table status file and can delete index files immediately. However, in legacy stores when alter segment merge is called, after merge index success, do not delete index files immediately as it may cause issues for parallel queries.

Issue Links

• is duplicated by: CARBONDATA-4145 Query fails and the message "File does not exist: xxxx.carbondata" is displayed
• is duplicated by: CARBONDATA-4146 Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.