Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2651

codegen overhead can be high

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.2, Impala 2.3.0
    • Fix Version/s: None
    • Component/s: Backend

      Description

      We received reports of excessive codegen compilation/optimization times for very large expressions generated by visualization tools.

      We should:

      1. Expose codegen optimization levels as query options. Currently there is only an all or nothing codegen query option. It's likely that overly complex expressions such as hundreds of cascading conditions take very long and benefit very little from an O2 optimization level, but they could still run significantly faster even at O0 or O1 versus interpreted.
      2. Consider dropping to O1 (or turn off riskier passes individually) automatically for very large expressions.
      3. Consider parameterizing the compilation duration time limits and set a reasonable default, say 10 seconds. Either disable codegen or reduce it to, say, O0 if compilation takes longer than the preset limit.

      Workaround
      In some cases disabling codegen can help.

      SET disable_codegen=true;
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                srus Silvius Rus
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: