Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-2136

Enhance cubing algorithm selection

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      To be more protective to select in-mem cubing.

      There were reports that in-mem cubing was selected but run slowly (even timeout) in a few small PoC clusters. Apart from the limited resource in PoC, another important cause is the algorithm selection didn't concern the scale of the job. When mappers are many but mapper slots are few, it will take many rounds to finish all the mappers, and the total time of in-mem cubing becomes slow.

      The scale of the job should be taken into consideration when auto selecting cubing algorithm. Namely a mapper limit configuration is introduced, to stop in-mem cubing when the number of mappers goes beyond the threshold.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                liyang.gmt8@gmail.com liyang
                Reporter:
                liyang.gmt8@gmail.com liyang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: