Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1566

Add and use a separate kylin_job_conf.xml for in-mem cubing

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.5.0
    • Fix Version/s: v1.5.2
    • Component/s: Job Engine
    • Labels:
      None

      Description

      From v1.5.0, kylin has two cubing algorithm: by-layer and by-split(in-mem).

      The in-mem cubing require more memory resource for the mapper; so far it is configured by parameter "kylin.job.cubing.inmem.mrjob_conf_override" with default value "mapreduce.map.java.opts=-Xmx2700m; mapreduce.map.memory.mb=3072; mapreduce.task.io.sort.mb=200"; This is a little confusing as user can also configure such parameters in kylin_job_conf.xml

      A better way is adding a new config file for in-mem cubing job, moving the in-mem specific mapreduce parameters there.

        Attachments

          Activity

            People

            • Assignee:
              shaofengshi Shao Feng Shi
              Reporter:
              shaofengshi Shao Feng Shi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: