[CARBONDATA-3834] Segment directory and segment file in metadata are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to false.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: hadoop-integration, spark-integration
• Assignee: Unassigned
• Reporter: Venugopal Reddy K
• Created: 27/May/20 13:48
• Updated: 06/Oct/20 07:06
• Resolved: 06/Oct/20 07:06

Description

[Issue]

With the latest version of Carbon, Segment directory and segment file in metadata directory are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to 'false'. And actual index files which were present in respective partition's '.tmp' directory are also deleted without moving them out to respective partition directory where its '.carbondata' file exist. Thus all the queries fails due to this problem.

Exception in thread "main" java.lang.NullPointerExceptionException in thread "main" java.lang.NullPointerException at org.apache.carbondata.core.readcommitter.TableStatusReadCommittedScope.getCommittedIndexFile(TableStatusReadCommittedScope.java:90) at org.apache.carbondata.core.index.Segment.getCommittedIndexFile(Segment.java:183) at org.apache.carbondata.core.util.BlockletIndexUtil.getTableBlockUniqueIdentifiers(BlockletIndexUtil.java:204) at 

 

This issue was introduced from the resolution of an older optimization issue CARBONDATA-3641[Should improve data loading performance for partition table]

i.e., with https://github.com/apache/carbondata/pull/3535

 

[Modification Suggestion]

 

If 'carbon.merge.index.in.segment' property is false, we can create the segment directory and segment file, and move the index file from respective partition's temp directory to partition directory where the .carbondata file exists.

Note: This need to be done before the respective partition's .tmp directory is deleted. Otherwise, we loose the index files.