Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5140

Fix CompileException in run-time generated code when record batch has large number of fields.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.9.0
    • Fix Version/s: 1.11.0
    • Component/s: Execution - Flow
    • Labels:
      None

      Description

      CTAS that does SELECT over 5003 columns fails with CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject...

      Drill 1.9.0 git commit ID : 4c1b420b
      CTAS statement and CSV data file are attached.

      I ran test with and without setting the below system option, test failed in both cases.
      alter system set `exec.java_compiler`='JDK';

      sqlline session just closes with below message, after the failing CTAS is executed.
      Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl

      Stack trace from drillbit.log

      2016-12-20 12:02:16,016 [27a6e241-99b1-1f2a-8a91-394f8166e969:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[ProjectorGen45.java]', Line 11, Column 8: ProjectorGen45.java:11: error: too many constants
      public class ProjectorGen45 {
             ^ (compiler.err.limit.pool)
      
      Fragment 0:0
      
      [Error Id: ced84dce-669d-47c2-b5d2-5e0559dbd9fd on centos-01.qa.lab:31010]
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[ProjectorGen45.java]', Line 11, Column 8: ProjectorGen45.java:11: error: too many constants
      public class ProjectorGen45 {
             ^ (compiler.err.limit.pool)
      
      Fragment 0:0
      
      [Error Id: ced84dce-669d-47c2-b5d2-5e0559dbd9fd on centos-01.qa.lab:31010]
              at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) ~[drill-common-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.9.0.jar:1.9.0]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
              at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
      Caused by: org.apache.drill.exec.exception.SchemaChangeException: Failure while attempting to load generated class
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:487) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:91) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_91]
              at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_91]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0-mapr-1607.jar:na]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226) [drill-java-exec-1.9.0.jar:1.9.0]
              ... 4 common frames omitted
      Caused by: org.apache.drill.exec.exception.ClassTransformationException: java.util.concurrent.ExecutionException: org.apache.drill.exec.exception.ClassTransformationException: Failure generating transformation classes for value:
      ...
      

      Thanks,
      Khurram

        Attachments

        1. drill_5117.q
          125 kB
          Khurram Faraaz
        2. manyColumns.csv
          19 kB
          Khurram Faraaz

          Issue Links

            Activity

              People

              • Assignee:
                vvysotskyi Volodymyr Vysotskyi
                Reporter:
                khfaraaz Khurram Faraaz
                Reviewer:
                Khurram Faraaz
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: