Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-1109

Page lost in load process when last page is not be consumed at the end

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0, 1.1.1
    • Component/s: data-load
    • Labels:
      None

      Description

      First, we use Producer-Consumer model in the write step, we have n(default value is 2 and can be configured) producers and one consumer. The task of generate last page(less than 32000) is added to thread pool at the end, but can't be guaranteed to be finished and add to BlockletDataHolder at the end. Because we have n tasks running concurrently.
      Second, we have 2 ways to invoke `writeDataToFile`, one is the size of `DataWriterHolder` reach the size of blocklet and two is the page is the last page.
      So if the last page is not be consumed at the end, we lost the page which be consumed after last page.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                waterman Yadong Qi
                Reporter:
                waterman Yadong Qi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h