Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3481

Multi-thread pruning fails when datamaps count is just near numOfThreadsForPruning

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.1
    • Component/s: None
    • Labels:
      None

      Description

      Problem : Multi-thread pruning fails when datamaps count is just near numOfThreadsForPruning.

      Cause : When the datamaps count is just near numOfThreadsForPruning,
      As code is checking '>= ', last thread may not get the datamaps for prune. Hence array out of index exception is thrown in this scenario.
      There is no issues with higher number of datamaps.

      solution: In this scenario launch threads based on the distribution value, not on the hardcoded value

       

      steps to reproduce:

      5 index files with total 0.32 million data file (each has 50k, 120k, 50k, 60k, 50k each)

      with default thread count as 4, array of bound index is observed for below line.

      final List<SegmentDataMapGroup> segmentDataMapGroups = datamapListForEachThread.get;

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Ajantha_Bhat Ajantha Bhat
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8h 10m
                  8h 10m