[CARBONDATA-2092] Fix compaction bug to prevent the compaction flow from going through the restructure compaction flow

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Assignee: Manish Gupta
• Reporter: Manish Gupta
• Created: 29/Jan/18 09:12
• Updated: 01/Feb/18 06:36
• Resolved: 01/Feb/18 06:36

Description

Problem and analysis:

----------------------------------------

During data load current schema timestamp is written to the carbondata fileHeader. This is used during compaction to decide whether the block is a restructured block or the block is according to the latest schema.

As the blocklet information is now stored in the index file, while laoding it in memory the carbondata file header is not read and due to this the schema timestamp is not getting set to the blocklet information. Due to this during compaction flow there is a mismatch on comparing the current schema time stamp with the timestamp stored in the block and the flow goes through the restructure compaction flow instead of normal compaction flow.

Impact:

-------------

Compaction performance degradation as restructure compaction flow involves sorting of data again.

Issue Links

• links to: GitHub Pull Request #1875