Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23580

Interpreted mode fallback should be implemented for all expressions & projections

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:

      Description

      Spark SQL currently does not support interpreted mode for all expressions and projections. This is a problem for scenario's where were code generation does not work, or blows past the JVM class limits. We currently cannot gracefully fallback.

      This ticket is an umbrella to fix this class of problem in Spark SQL. This work can be divided into two main area's:

      • Add interpreted versions for all dataset related expressions.
      • Add an interpreted version of GenerateUnsafeProjection.

        Attachments

        1.
        Add an interpreted version of GenerateUnsafeProjection Sub-task Resolved Herman van Hovell
        2.
        Add interpreted execution to StaticInvoke expression Sub-task Resolved Kazuaki Ishizaki
        3.
        Add interpreted execution to Invoke expression Sub-task Resolved Kazuaki Ishizaki
        4.
        Add interpreted execution for UnwrapOption expression Sub-task Resolved Marco Gaido
        5.
        Add interpreted execution to NewInstance expression Sub-task Resolved Takeshi Yamamuro
        6.
        Add interpreted execution for WrapOption expression Sub-task Resolved Marco Gaido
        7.
        Add interpreted execution for MapObjects expression Sub-task Resolved Liang-Chi Hsieh
        8.
        Add interpreted execution for CatalystToExternalMap expression Sub-task Resolved Takeshi Yamamuro
        9.
        Add interpreted execution for ExternalMapToCatalyst expression Sub-task Resolved Takeshi Yamamuro
        10.
        Add interpreted execution for CreateExternalRow expression Sub-task Resolved Marco Gaido
        11.
        Add interpreted execution for EncodeUsingSerializer expression Sub-task Resolved Marco Gaido
        12.
        Add interpreted execution for DecodeUsingSerializer expression Sub-task Resolved Marco Gaido
        13.
        Add interpreted execution for InitializeJavaBean expression Sub-task Resolved Liang-Chi Hsieh
        14.
        Add interpreted execution for GetExternalRowField expression Sub-task Resolved Takeshi Yamamuro
        15.
        Add interpreted execution for ValidateExternalType expression Sub-task Resolved Takeshi Yamamuro
        16.
        Modify Dataset test harness to include interpreted execution Sub-task Resolved Liang-Chi Hsieh
        17.
        Audit Spark SQL code base for non-interpreted expressions Sub-task Resolved Unassigned
        18.
        Extend ExpressionEvalHelper harness to also test failures Sub-task Resolved Takeshi Yamamuro
        19.
        Add fallback to interpreted execution logic Sub-task Resolved Liang-Chi Hsieh
        20.
        Investigate replacing Code Generated UnsafeRowJoiner with an Interpreted version Sub-task In Progress Herman van Hovell
        21.
        Clean-up UnsafeWriter classes Sub-task Resolved Kazuaki Ishizaki
        22.
        Add Unsafe* copy methods to UnsafeWriter Sub-task Resolved Herman van Hovell
        23.
        Add interpreted execution to SortPrefix expression Sub-task Resolved Unassigned
        24.
        Add interpreted execution to SortPrefix expression Sub-task Resolved Bruce Robbins
        25.
        MutableProjection supports fallback to an interpreted mode Sub-task Resolved Takeshi Yamamuro
        26.
        SafeProjection supports fallback to an interpreted mode Sub-task Resolved Takeshi Yamamuro

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hvanhovell Herman van Hovell
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated: