Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3834

Segment directory and segment file in metadata are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to false.

    XMLWordPrintableJSON

    Details

      Description

      [Issue]

      With the latest version of Carbon, Segment directory and segment file in metadata directory are not created for partitioned table when 'carbon.merge.index.in.segment' property is set to 'false'. And actual index files which were present in respective partition's '.tmp' directory are also deleted without moving them out to respective partition directory where its '.carbondata' file exist. Thus all the queries fails due to this problem.

      Exception in thread "main" java.lang.NullPointerExceptionException in thread "main" java.lang.NullPointerException at org.apache.carbondata.core.readcommitter.TableStatusReadCommittedScope.getCommittedIndexFile(TableStatusReadCommittedScope.java:90) at org.apache.carbondata.core.index.Segment.getCommittedIndexFile(Segment.java:183) at org.apache.carbondata.core.util.BlockletIndexUtil.getTableBlockUniqueIdentifiers(BlockletIndexUtil.java:204) at 

       

      This issue was introduced from the resolution of an older optimization issue CARBONDATA-3641[Should improve data loading performance for partition table]

      i.e., with https://github.com/apache/carbondata/pull/3535

       

      [Modification Suggestion]

       

      If 'carbon.merge.index.in.segment' property is false, we can create the segment directory and segment file, and move the index file from respective partition's temp directory to partition directory where the .carbondata file exists.

      Note: This need to be done before the respective partition's .tmp directory is deleted. Otherwise, we loose the index files.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              VenuReddy Venugopal Reddy K
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 8h
                8h