Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-4037

Improve the table status and segment file writing

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.2.0
    • None
    • None

    Description

      Currently, we update table status and segment files multiple times for a single iud/merge/compact operation and delete the index files immediately after merge. When concurrent queries are run, there may be situations like user query is trying to access the segment index files and they are not present, which is availability issue.

      • To solve above issue, we can make mergeindex files generation mandatory and fail load/compaction if mergeindex fails. Then if merge index is success, update table status file and can delete index files immediately. However, in legacy stores when alter segment merge is called, after merge index success, do not delete index files immediately as it may cause issues for parallel queries.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            Shreelekhya SHREELEKHYA GAMPA
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 27.5h
                27.5h

                Slack

                  Issue deployment