Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4200

SparkCubingBylayer is not rubust using CubeStatsReader to estimateLayerPartitionNum

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: v2.6.4
    • Fix Version/s: None
    • Component/s: Job Engine
    • Labels:
      None

      Description

      In prod env, I found some scene as follow:
      if user has measure of Bitmap, the Spark tasks often block at job 0,1 which compute layer 0,1.
      after analyze spark log, I found that, spark use CubeStatsReader to estimateLayerPartition Num, If layer 0 has cuboid 255, and its size is 10M, according to the default configuration pararm ,the partition will be 1, however it's quick small for this amout of data.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              aahi hailin.huang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: