Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41283

Feature parity: Functions API in Spark Connect

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Critical
    • Resolution: Done
    • 3.4.0
    • 3.4.0
    • Connect
    • None

    Description

      Implement functions API.

      Attachments

        Issue Links

          split from

          Sub-task - The sub-task of the issue SPARK-39674 Initial protobuf definition for Spark Connect API

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #39698 [Github] Pull Request #39698 (zhengruifeng)

          1.
          		 Add missing PySpark functions to Spark Connect Sub-task Resolved Martin Grund
          2.
          		 Homogenize built-in function names for binary operators like + - / Sub-task Resolved Unassigned
          3.
          		 Disable unsupported functions Sub-task Resolved Martin Grund
          4.
          		 Extend type support for Spark Connect literals Sub-task Resolved Martin Grund
          5.
          		 Support more datatypes Sub-task Resolved Ruifeng Zheng
          6.
          		 Make Literal support more datatypes Sub-task Resolved Ruifeng Zheng
          7.
          		 Implement asc and desc methods Sub-task Resolved Ruifeng Zheng
          8.
          		 Implement math functions Sub-task Resolved Ruifeng Zheng
          9.
          		 implement normal functions Sub-task Resolved Ruifeng Zheng
          10.
          		 implement `broadcast` function Sub-task Resolved Ruifeng Zheng
          11.
          		 Implement aggregation functions Sub-task Resolved Ruifeng Zheng
          12.
          		 Implement count_distinct and sum_distinct functions Sub-task Resolved Ruifeng Zheng
          13.
          		 implement `product` function Sub-task Resolved Ruifeng Zheng
          14.
          		 Implement part of string/binary functions Sub-task Resolved Xinrong Meng
          15.
          		 Implement date/timestamp functions Sub-task Resolved Xinrong Meng
          16.
          		 Support LambdaFunction expresssion Sub-task Resolved Ruifeng Zheng
          17.
          		 Implement `collection` functions: A~C Sub-task Resolved Ruifeng Zheng
          18.
          		 Implement the rest of string/binary functions Sub-task Resolved Xinrong Meng
          19.
          		 Implement `functions.format_number` Sub-task Resolved Ruifeng Zheng
          20.
          		 Implement `collection` functions: E~M Sub-task Resolved Ruifeng Zheng
          21.
          		 implement MISC function Sub-task Resolved Ruifeng Zheng
          22.
          		 Make csv functions support options Sub-task Resolved Ruifeng Zheng
          23.
          		 Implement `collection` functions: P~Z Sub-task Resolved Ruifeng Zheng
          24.
          		 Implement Partition Transformation Functions Sub-task Resolved Ruifeng Zheng
          25.
          		 Implement json functions Sub-task Resolved Ruifeng Zheng
          26.
          		 implement `Window` functions Sub-task Resolved Ruifeng Zheng
          27.
          		 Implement the rest of Lambda functions Sub-task Resolved Ruifeng Zheng
          28.
          		 Enable the deprecated functions Sub-task Resolved Ruifeng Zheng
          29.
          		 Remove `FunctionBuilder` Sub-task Resolved Ruifeng Zheng
          30.
          		 Implement time window functions Sub-task Resolved Ruifeng Zheng
          31.
          		 Implement `sequence` function Sub-task Resolved Ruifeng Zheng
          32.
          		 Implement `call_udf` function Sub-task Resolved Ruifeng Zheng
          33.
          		 Implement `unwrap_udt` function Sub-task Resolved Ruifeng Zheng
          34.
          		 Deduplicate helper functions Sub-task Resolved Ruifeng Zheng
          35.
          		 Enable test for `array_repeat` Sub-task Resolved Ruifeng Zheng
          36.
          		 Make json functions support options Sub-task Resolved Ruifeng Zheng
          37.
          		 Make json functions support DataType Schema Sub-task Resolved Ruifeng Zheng
          38.
          		 log() function variations are missing Sub-task Resolved Ruifeng Zheng
          39.
          		 Implement `intX2` function Sub-task Resolved Unassigned
          40.
          		 Fix `nanvl` function Sub-task Resolved Ruifeng Zheng
          41.
          		 Fix `pmod` function Sub-task Resolved Ruifeng Zheng
          42.
          		 Resolve dtypes inconsistencies of date/timestamp functions Sub-task Resolved Ruifeng Zheng
          43.
          		 Add the unsupported list for functions Sub-task Resolved Ruifeng Zheng
          44.
          		 Add function array_compact Sub-task Resolved Ruifeng Zheng
          45.
          		 Make json functions support options Sub-task Resolved Ruifeng Zheng
          46.
          		 Fix `count(expr("*"))` function Sub-task Resolved Ruifeng Zheng
          47.
          		 Implement `transform_values` function Sub-task Resolved Ruifeng Zheng
          48.
          		 Implement `transform_keys` function Sub-task Resolved Ruifeng Zheng
          49.
          		 Function `from_json` should support non-literal expression Sub-task Resolved Ruifeng Zheng
          50.
          		 Map data show in different order Sub-task Resolved Ruifeng Zheng
          51.
          		 Literal should support numpy datatypes Sub-task Resolved Ruifeng Zheng
          52.
          		 Different column name of lit(np.int8) Sub-task Resolved Takuya Ueshin
          53.
          		 Literal should throw TypeError for unsupported DataType Sub-task Resolved Ruifeng Zheng
          54.
          		 Make `count(*)` work correctly Sub-task Resolved Ruifeng Zheng
          55.
          		 Add `array_append` to Connect Sub-task Resolved Ruifeng Zheng
          56.
          		 Add `array_insert` to Connect Sub-task Resolved Ruifeng Zheng
          57.
          		 `array_append` should accept `Any` value Sub-task Resolved Ruifeng Zheng
          58.
          		 Enable more parity tests related to functions Sub-task Resolved Takuya Ueshin
          59.
          		 Implement Avro functions Sub-task Open Unassigned
          60.
          		 Python UDF with UDT Sub-task Resolved Takuya Ueshin

          Activity

            People

              podongfeng Ruifeng Zheng
              gurwls223 Hyukjin Kwon
              Hyukjin Kwon Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: