Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6763

Codegen optimization of SQL functions with constant values

    XMLWordPrintableJSON

    Details

      Description

      Codegen class compilation takes tens to hundreds of milliseconds, a class cache is hit when generifiedCode of code generator is exactly the same.
      It works fine when UDF only takes columns or symbols, but not efficient when one or more parameters in UDF is always distinct from the other.
      Take face recognition for example, the face images are almost distinct from each other according to lighting, facial expressions and details.
      It is important to reduce redundant class compilation especially for those low latency queries.
      Cache miss rate and metaspace gc can also be reduced by eliminating the redundant classes.

      Here is the query to get the persons whose last name is Brunner and hire from 1st Jan 1990:
      SELECT full_name, hire_date FROM cp.`employee.json` where last_name = 'Brunner' and hire_date >= '1990-01-01 00:00:00.0';
      Now get the persons whose last name is Bernard and hire from 1st Jan 1990.
      SELECT full_name, hire_date FROM cp.`employee.json` where last_name = 'Bernard' and hire_date >= '1990-01-01 00:00:00.0';

      Figure shows the compilation time of the generated code by the above query in FilterRecordBatch on my laptop
      Figure  shows the only difference of the generated code from the attachments is the last_name value at line 156.
      It is straightforward that the redundant class compilation can be eliminated by making the string12 as a member of the class and set the value when the instance is created

        Attachments

        1. Query2.java
          6 kB
          shuifeng lu
        2. Query1.java
          6 kB
          shuifeng lu
        3. compilation_time.png
          63 kB
          shuifeng lu
        4. code_compare.png
          129 kB
          shuifeng lu

          Issue Links

            Activity

              People

              • Assignee:
                shuifeng shuifeng lu
                Reporter:
                shuifeng shuifeng lu
                Reviewer:
                Vova Vysotskyi
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: