Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-2258

Separate visible and invisible segments info into two files to reduce the size of tablestatus file.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.3.2
    • Fix Version/s: 1.4.0, 1.3.2
    • Component/s: core
    • Labels:
      None

      Description

      The size of the tablestatus file is getting larger, there are many places will scan this file and it will impact the performance of reading this file.
      According to the discussion on thread, it can append the
      invisible segment list to the file called 'tablestatus.history' when execute
      command 'CLEAN FILES FOR TABLE' (in method 'SegmentStatusManager.deleteLoadsAndUpdateMetadata') every time, separate visible and invisible segments into two files(tablestatus file and tablestatus.history file).
      If later it needs to support listing all segments(include visible and invisible) list when execute 'SHOW SEGMENTS FOR TABLE', it just need to read from two files.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zzcclp Zhichao Zhang
                Reporter:
                zzcclp Zhichao Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5h 50m
                  5h 50m