Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-2866

Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: v2.2.0
    • Fix Version/s: None
    • Component/s: Job Engine
    • Labels:
      None

      Description

      Currently only one reducer is assigned for hll stats calculation, which may become the bottleneck for slow down this step. Since the stats for different cuboids will not influence each other, it's better to divide the cuboid set into several and assign a reduce for each subset.
      The strategy of this patch is to assign 100 cuboids into a subset. And there's a upper limit of reducers for hll stats calculation. Currently it's 50.

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            yaho Zhong Yanghong
            Reporter:
            yaho Zhong Yanghong
            Request participants:
            None
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated: