Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1861

Hive may fail to create flat table with "GC overhead error"

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.5.3
    • Component/s: None
    • Labels:
      None

      Description

      in conf/kylin_hive_conf.xml, it enables "hive.auto.convert.join.noconditionaltask", which is the optimization about converting common join into mapjoin based on the input file size, and give a big value (300000000, 300MB) to "hive.auto.convert.join.noconditionaltask.size". So if the memory of the mapper isn't enough, it will cause the error.

      To fix the error, user can reduce the "hive.auto.convert.join.noconditionaltask.size" in conf/kylin_hive_conf.xml, and then retry the job.

        Attachments

          Activity

            People

            • Assignee:
              shaofengshi Shao Feng Shi
              Reporter:
              shaofengshi Shao Feng Shi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: