Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31936

Implement ScriptTransform in sql/core

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: 3.0.0, 3.1.0, 3.2.0
    • Fix Version/s: 3.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      ScriptTransformation currently relies on Hive internals. It'd be great if we can implement a native ScriptTransformation in sql/core module to remove the extra Hive dependency here.

        Attachments

        1.
        Spark can’t support TRANSFORM with aggregation Sub-task Resolved angerszhu
        2.
        Support processing array/map/struct type using spark noserde mode Sub-task Resolved angerszhu
        3.
        To perfect using constraints of script transform Sub-task Resolved Unassigned
        4.
        Solve string value error about Date/Timestamp in ScriptTransform Sub-task Resolved Unassigned
        5.
        Refactor current script transform code Sub-task Resolved angerszhu
        6.
        Implement script transform in sql/core Sub-task Resolved angerszhu
        7.
        TRANSFORM when schema less should keep same with hive Sub-task Resolved angerszhu
        8.
        TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType Sub-task Resolved Unassigned
        9.
        Test coverage of HiveScripTransformationExec Sub-task Resolved angerszhu
        10.
        SCRIP TRANSFORM Extract common method from process row to avoid repeated judgement Sub-task Resolved angerszhu
        11.
        Catch and adding excetion about HiveInspector can't converted data type Sub-task Resolved Unassigned
        12.
        Script Transformation no-serde `TOK_TABLEROWFORMATLINES` only support `\n` Sub-task Resolved angerszhu
        13.
        Script Transform DELIMIT value should be formatted Sub-task Resolved angerszhu
        14.
        Scrip transformation no-serde mode when column less then output length , Use null fill Sub-task Resolved angerszhu
        15.
        update "no-serde" in the codebase in other TRANSFORM PRs. Sub-task Closed Unassigned
        16.
        Add a test case for hive serde/default-serde mode's null value '\\N' Sub-task Resolved angerszhu
        17.
        Script transform hive serde default field.delimit is '\t' Sub-task Resolved angerszhu
        18.
        Add a configuration to control the legacy behavior of whether need to pad null value when value size less then schema size Sub-task Resolved angerszhu
        19.
        Spark SQL no serde row format field delimit default is '\u0001' Sub-task Resolved angerszhu
        20.
        Support user-defined script command wrapper for more use case Sub-task Resolved angerszhu
        21.
        Add a dedicated SQL document page for the TRANSFORM-related functionality, Sub-task Resolved angerszhu
        22.
        Transform with clusterby/orderby/sortby Sub-task Resolved angerszhu
        23.
        Refactor ScriptTransformation to remove input parameter and replace it by child.output Sub-task Resolved angerszhu
        24.
        TRANSFORM forbiden DISTINCT/ALL and make the error message clear Sub-task Resolved angerszhu
        25.
        TRANSFORM should not support ALIAS in input expression seq Sub-task Resolved angerszhu
        26.
        extract doc of hive format Sub-task Resolved angerszhu
        27.
        DayTimeIntervalType/YearMonthIntervalString show different in hive serde and row formet delimited Sub-task Resolved angerszhu
        28.
        Add SQL doc about transform for current behavior Sub-task Resolved angerszhu
        29.
        Row FORMAT SERDE should handle null value as `\N` Sub-task Resolved angerszhu

          Activity

            People

            • Assignee:
              angerszhuuu angerszhu
              Reporter:
              angerszhuuu angerszhu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: