Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5164

Equi-join query results in CompileException when inputs have large number of columns

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.9.0
    • Fix Version/s: 1.11.0
    • Component/s: Execution - Codegen
    • Labels:
      None

      Description

      Drill 1.9.0
      git commit ID : 4c1b420b
      4 node CentOS cluster
      JSON file has 4095 keys (columns)

      0: jdbc:drill:schema=dfs.tmp> select * from `manyColsInJson.json` t1, `manyColsInJson.json` t2 where t1.key2000 = t2.key2000;
      Error: SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]', Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
          public void doSetup(FragmentContext context, VectorContainer buildBatch, RecordBatch probeBatch, RecordBatch outgoing)
                      ^ (compiler.err.limit.code)
      
      Fragment 0:0
      
      [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010] (state=,code=0)
      

      Stack trace from drillbit.log

      2016-12-26 09:52:11,321 [279f17fd-c8f0-5d18-1124-76099f0a5cc8:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]', Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
          public void doSetup(FragmentContext context, VectorContainer buildBatch, RecordBatch probeBatch, RecordBatch outgoing)
                      ^ (compiler.err.limit.code)
      
      Fragment 0:0
      
      [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010]
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]', Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
          public void doSetup(FragmentContext context, VectorContainer buildBatch, RecordBatch probeBatch, RecordBatch outgoing)
                      ^ (compiler.err.limit.code)
      
      Fragment 0:0
      
      [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010]
              at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) ~[drill-common-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.9.0.jar:1.9.0]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
              at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
      Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: org.apache.drill.exec.exception.SchemaChangeException: org.apache.drill.exec.exception.ClassTransformationException: java.util.concurrent.ExecutionException: org.apache.drill.exec.exception.ClassTransformationException: Failure generating transformation classes for value:
      
      package org.apache.drill.exec.test.generated;
      
      ...
      public class HashJoinProbeGen294 {
      
          NullableVarCharVector[] vv0;
          NullableVarCharVector vv3;
          NullableVarCharVector[] vv6;
                  ...
                  vv49137 .copyFromSafe((probeIndex), (outIndex), vv49134);
                  vv49143 .copyFromSafe((probeIndex), (outIndex), vv49140);
                  vv49149 .copyFromSafe((probeIndex), (outIndex), vv49146);
              }
          }
          
      public void __DRILL_INIT__()
              throws SchemaChangeException
          {
          }
      
      }
      
              at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:78) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:74) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
              at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
              ... 34 common frames omitted
      Caused by: org.codehaus.commons.compiler.CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]', Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
          public void doSetup(FragmentContext context, VectorContainer buildBatch, RecordBatch probeBatch, RecordBatch outgoing)
                      ^ (compiler.err.limit.code)
              at org.apache.drill.exec.compile.DrillDiagnosticListener.report(DrillDiagnosticListener.java:43) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at com.sun.tools.javac.api.ClientCodeWrapper$WrappedDiagnosticListener.report(ClientCodeWrapper.java:559) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.util.Log.writeDiagnostic(Log.java:428) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.util.Log.report(Log.java:416) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.util.AbstractLog.error(AbstractLog.java:94) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.jvm.Code.checkLimits(Code.java:95) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.jvm.Gen.genMethod(Gen.java:948) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.jvm.Gen.visitMethodDef(Gen.java:885) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.tree.JCTree$JCMethodDecl.accept(JCTree.java:669) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.jvm.Gen.genDef(Gen.java:684) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.jvm.Gen.genClass(Gen.java:2232) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.main.JavaCompiler.genCode(JavaCompiler.java:712) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.main.JavaCompiler.generate(JavaCompiler.java:1451) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.main.JavaCompiler.generate(JavaCompiler.java:1419) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.main.JavaCompiler.compile2(JavaCompiler.java:870) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:829) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.main.Main.compile(Main.java:439) ~[jdk.tools-1.7.jar:na]
              at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:132) ~[jdk.tools-1.7.jar:na]
              at org.apache.drill.exec.compile.JDKClassCompiler.getByteCode(JDKClassCompiler.java:71) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.compile.AbstractClassCompiler.getClassByteCode(AbstractClassCompiler.java:53) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.compile.QueryClassLoader$ClassCompilerSelector.getClassByteCode(QueryClassLoader.java:143) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.compile.QueryClassLoader$ClassCompilerSelector.access$100(QueryClassLoader.java:111) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode(QueryClassLoader.java:104) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:224) ~[drill-java-exec-1.9.0.jar:1.9.0]
              ... 38 common frames omitted
      2016-12-26 09:52:12,251 [CONTROL-rpc-event-queue] WARN  o.a.drill.exec.work.foreman.Foreman - Dropping request to move to COMPLETED state as query is already at FAILED state (which is terminal).
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vvysotskyi Volodymyr Vysotskyi
                Reporter:
                khfaraaz Khurram Faraaz
                Reviewer:
                Jinfeng Ni
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: