Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3367

OOM when huge number of carbondata files are read from SDK reader

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, for each carbondata file, one CarbonRecordReader will be created. And list of CarbonRecordReader will be maintained in carbonReader. so even when CarbonRecordReader is closed, the GC will not happen for that reader as list is still referring that object. 

      so, each CarbonRecordReader needs separate memory , instead of reusing the previous memory. 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Ajantha_Bhat Ajantha Bhat
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 40m
                  2h 40m