Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1566

Add and use a separate kylin_job_conf.xml for in-mem cubing

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • v1.5.0
    • v1.5.2
    • Job Engine
    • None

    Description

      From v1.5.0, kylin has two cubing algorithm: by-layer and by-split(in-mem).

      The in-mem cubing require more memory resource for the mapper; so far it is configured by parameter "kylin.job.cubing.inmem.mrjob_conf_override" with default value "mapreduce.map.java.opts=-Xmx2700m; mapreduce.map.memory.mb=3072; mapreduce.task.io.sort.mb=200"; This is a little confusing as user can also configure such parameters in kylin_job_conf.xml

      A better way is adding a new config file for in-mem cubing job, moving the in-mem specific mapreduce parameters there.

      Attachments

        Activity

          People

            shaofengshi Shao Feng Shi
            shaofengshi Shao Feng Shi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: