Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3678

Plan generating for Drill on Hive takes huge java heap size

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.1.0
    • Future
    • Query Planning & Optimization
    • None

    • 1.1

    Description

      ===Env===
      Drill 1.0 on Hive 0.13
      (Also tested Drill 1.1 and get the same behavior)

      8 nodes drill cluster.
      Jave heap size is set to 8G and Direct memory is set to 96G on each drillbit.

      ===Symptom===
      This is a Hive parquet partition table which has multi-level partitions.
      The Hive table size is several TB with tens of thousands leaf partitions.

      When doing a "select * from table limit 10", the query keeps in "pending" state to generate the SQL plan. And finally the drillbits crashed with java heap OOM.

      java.lang.OutOfMemoryError: Java heap space
        at hive.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:599)
        at hive.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:360)
        at hive.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:100)
        at hive.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:172)
        at hive.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:130)
        at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
        at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:66)
        at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
        at org.apache.drill.exec.store.hive.HiveRecordReader.init(HiveRecordReader.java:246)
        at org.apache.drill.exec.store.hive.HiveRecordReader.<init>(HiveRecordReader.java:138)
        at org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:58)
        at org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:34)
        at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:150)
        at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:173)
        at org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:106)
        at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:81)
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:235)
        at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Node ran out of Heap memory, exiting.
java.lang.OutOfMemoryError: Java heap space

      We captured some stacktraces of foreman thread in foreman drillbit, here are 2 times' examples:

      2a482cd9-7fb2-c492-1356-d049e90870c8:foreman id=115 state=RUNNABLE
    at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeData(Unknown Source)
    at org.apache.xerces.dom.ElementImpl.getTagName(Unknown Source)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2348)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2234)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2151)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:871)
    at org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2069)
    at org.apache.hadoop.mapred.JobConf.(JobConf.java:421)
    at org.apache.drill.exec.store.hive.HiveScan.splitInput(HiveScan.java:178)
    at org.apache.drill.exec.store.hive.HiveScan.getSplits(HiveScan.java:167)
    at org.apache.drill.exec.store.hive.HiveScan.access$000(HiveScan.java:69)
    at org.apache.drill.exec.store.hive.HiveScan$1.run(HiveScan.java:146)
    at org.apache.drill.exec.store.hive.HiveScan$1.run(HiveScan.java:144)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
    at org.apache.drill.exec.store.hive.HiveScan.getSplitsWithUGI(HiveScan.java:144)
    at org.apache.drill.exec.store.hive.HiveScan.(HiveScan.java:119)
    at org.apache.drill.exec.store.hive.HiveStoragePlugin.getPhysicalScan(HiveStoragePlugin.java:78)
    at org.apache.drill.exec.store.hive.HiveStoragePlugin.getPhysicalScan(HiveStoragePlugin.java:41)
    at org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:53)
    at org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:72)
    at org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:81)
    at org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:63)
    at org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:37)
    at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
    at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
    at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
    at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:316)
    at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:514)
    at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:211)
    at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:246)
    at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:162)
    at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
    at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:903)
    at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:242)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

--------

2a482cd9-7fb2-c492-1356-d049e90870c8:foreman id=115 state=RUNNABLE
    at java.util.zip.Inflater.inflateBytes(Native Method)
    at java.util.zip.Inflater.inflate(Inflater.java:259)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
      - locked java.io.InputStreamReader@63f0c79a
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:154)
    at java.io.BufferedReader.readLine(BufferedReader.java:317)
      - locked java.io.InputStreamReader@63f0c79a
    at java.io.BufferedReader.readLine(BufferedReader.java:382)
    at javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:319)
    at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
    at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2257)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2234)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2151)
      - locked org.apache.hadoop.mapred.JobConf@6ef80e0e
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:871)
    at org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2069)
    at org.apache.hadoop.mapred.JobConf.(JobConf.java:421)
    at org.apache.drill.exec.store.hive.HiveScan.splitInput(HiveScan.java:178)
    at org.apache.drill.exec.store.hive.HiveScan.getSplits(HiveScan.java:167)
    at org.apache.drill.exec.store.hive.HiveScan.access$000(HiveScan.java:69)
    at org.apache.drill.exec.store.hive.HiveScan$1.run(HiveScan.java:146)
    at org.apache.drill.exec.store.hive.HiveScan$1.run(HiveScan.java:144)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
    at org.apache.drill.exec.store.hive.HiveScan.getSplitsWithUGI(HiveScan.java:144)
    at org.apache.drill.exec.store.hive.HiveScan.(HiveScan.java:119)
    at org.apache.drill.exec.store.hive.HiveStoragePlugin.getPhysicalScan(HiveStoragePlugin.java:78)
    at org.apache.drill.exec.store.hive.HiveStoragePlugin.getPhysicalScan(HiveStoragePlugin.java:41)
    at org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:53)
    at org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:72)
    at org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:81)
    at org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:63)
    at org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:37)
    at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
    at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
    at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
    at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:316)
    at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:514)
    at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:211)
    at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:246)
    at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:162)
    at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
    at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:903)
    at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:242)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

      ===Workaround===
      We have to increase the JAVA heap size to 24GB and then query can finish successfully.
      Even though the query took about 16mins to generate the plan.

      ===Asks===
      Drill on Hive parquet query should utilize direct memory instead of java heap.

      Attachments

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. DRILL-3209 [Umbrella] Plan reads of Hive tables as native Drill reads when a native reader for the underlying table format exists

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              haozhu Hao Zhu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: