Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5140

Fix CompileException in run-time generated code when record batch has large number of fields.

Attach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.9.0
    • 1.11.0
    • Execution - Flow
    • None

    Description

      CTAS that does SELECT over 5003 columns fails with CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject...

      Drill 1.9.0 git commit ID : 4c1b420b
      CTAS statement and CSV data file are attached.

      I ran test with and without setting the below system option, test failed in both cases.
      alter system set `exec.java_compiler`='JDK';

      sqlline session just closes with below message, after the failing CTAS is executed.
      Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl

      Stack trace from drillbit.log

      2016-12-20 12:02:16,016 [27a6e241-99b1-1f2a-8a91-394f8166e969:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[ProjectorGen45.java]', Line 11, Column 8: ProjectorGen45.java:11: error: too many constants
      public class ProjectorGen45 {
             ^ (compiler.err.limit.pool)
      
      Fragment 0:0
      
      [Error Id: ced84dce-669d-47c2-b5d2-5e0559dbd9fd on centos-01.qa.lab:31010]
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: CompileException: File 'org.apache.drill.exec.compile.DrillJavaFileObject[ProjectorGen45.java]', Line 11, Column 8: ProjectorGen45.java:11: error: too many constants
      public class ProjectorGen45 {
             ^ (compiler.err.limit.pool)
      
      Fragment 0:0
      
      [Error Id: ced84dce-669d-47c2-b5d2-5e0559dbd9fd on centos-01.qa.lab:31010]
              at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) ~[drill-common-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) [drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.9.0.jar:1.9.0]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
              at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
      Caused by: org.apache.drill.exec.exception.SchemaChangeException: Failure while attempting to load generated class
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:487) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:91) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) ~[drill-java-exec-1.9.0.jar:1.9.0]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_91]
              at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_91]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0-mapr-1607.jar:na]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226) [drill-java-exec-1.9.0.jar:1.9.0]
              ... 4 common frames omitted
      Caused by: org.apache.drill.exec.exception.ClassTransformationException: java.util.concurrent.ExecutionException: org.apache.drill.exec.exception.ClassTransformationException: Failure generating transformation classes for value:
      ...
      

      Thanks,
      Khurram

      Attachments

        1. drill_5117.q
          125 kB
          Khurram Faraaz
        2. manyColumns.csv
          19 kB
          Khurram Faraaz

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            volodymyr Vova Vysotskyi
            khfaraaz Khurram Faraaz
            Khurram Faraaz Khurram Faraaz
            Votes:
            0 Vote for this issue
            Watchers:
            4 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment