Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41282

Feature parity: Column API in Spark Connect

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Reopened
    • Critical
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Connect
    • None

    Description

      Implement Column API.

      Attachments

        Issue Links

          split from

          Sub-task - The sub-task of the issue SPARK-39674 Initial protobuf definition for Spark Connect API

          • Major - Major loss of function.
          • Resolved
          1.
          		 Refactor "Column" for API Compatibility Sub-task Resolved Rui Wang
          2.
          		 Clean up unused QualifiedAttribute in Expression.proto Sub-task Resolved Rui Wang
          3.
          		 Support Column Alias in connect DSL Sub-task Resolved Rui Wang
          4.
          		 Rename `ColumnRef` to `Column` in Python client implementation Sub-task Resolved Rui Wang
          5.
          		 Window-function support Sub-task Resolved Ruifeng Zheng
          6.
          		 when-otherwise support Sub-task Resolved Ruifeng Zheng
          7.
          		 Add logical and string API to Column Sub-task Resolved Rui Wang
          8.
          		 Solve circular import between Column and _typing/functions Sub-task Resolved Hyukjin Kwon
          9.
          		 Fix `nullOrdering` in `SortOrder` Sub-task Resolved Ruifeng Zheng
          10.
          		 move SortField from relations.proto to expressions.proto Sub-task Resolved Ruifeng Zheng
          11.
          		 Support IsNull and IsNotNull in Column Sub-task Resolved Rui Wang
          12.
          		 Column does not support != Sub-task Resolved Martin Grund
          13.
          		 Refactor column related tests to test_connect_column Sub-task Resolved Rui Wang
          14.
          		 Implement `Cast` Sub-task Resolved Rui Wang
          15.
          		 Correctly infer the datatype of literal integers Sub-task Resolved Ruifeng Zheng
          16.
          		 Refactor LiteralExpression to support DataType Sub-task Resolved Ruifeng Zheng
          17.
          		 Implement `Column.isin` Sub-task Resolved Ruifeng Zheng
          18.
          		 Implement `Column.over` Sub-task Resolved Ruifeng Zheng
          19.
          		 Implement `Column.astype` Sub-task Resolved Ruifeng Zheng
          20.
          		 Make column op support `datetime` Sub-task Resolved Ruifeng Zheng
          21.
          		 Move Expressions to expressions.py Sub-task Resolved Ruifeng Zheng
          22.
          		 Make column op support `decimal` Sub-task Resolved Ruifeng Zheng
          23.
          		 Add invalid ops Sub-task Resolved Ruifeng Zheng
          24.
          		 Combine NullType and typed_null Sub-task Resolved Ruifeng Zheng
          25.
          		 Implement `Column.between` Sub-task Resolved Ruifeng Zheng
          26.
          		 Implement the column accessor Sub-task Resolved Ruifeng Zheng
          27.
          		 Implement `Column.name` Sub-task Resolved Ruifeng Zheng
          28.
          		 Support Column.bitwiseAND,bitwiseOR,bitwiseXOR,eqNullSafe,isNotNull,isNull,isin Sub-task Resolved Ruifeng Zheng
          29.
          		 Support Column.getField, Column.getItem Sub-task Resolved Ruifeng Zheng
          30.
          		 Fix arithmetic ops: negate, pow Sub-task Resolved Ruifeng Zheng
          31.
          		 Make the internal string op name consistent with FunctionRegistry Sub-task Resolved Ruifeng Zheng
          32.
          		 Implement `Column.{withField, dropFields}` Sub-task Resolved Ruifeng Zheng
          33.
          		 Support Window.rowsBetween Sub-task Resolved Ruifeng Zheng
          34.
          		 eqNullSafe does not support None as its argument Sub-task Resolved Ruifeng Zheng
          35.
          		 __getitem__ does not work with Column.isin Sub-task Resolved Ruifeng Zheng
          36.
          		 Window.partitionBy is not respected with row_number Sub-task Resolved Ruifeng Zheng
          37.
          		 Make column op support None Sub-task Resolved Ruifeng Zheng
          38.
          		 Add missing `__rmod__` Sub-task Resolved Ruifeng Zheng
          39.
          		 Column.eqNullSafe fails on NaN comparison Sub-task Resolved Ruifeng Zheng
          40.
          		 Column.isNull returns nan instead of None Sub-task Resolved Ruifeng Zheng
          41.
          		 Compatibility of string representation in Column Sub-task Resolved Hyukjin Kwon
          42.
          		 Enable pyspark.sql.connect.column.Column.withField doctest Sub-task Resolved Ruifeng Zheng
          43.
          		 Make `Column.__neg__` return the same column name as PySpark Sub-task Resolved Ruifeng Zheng
          44.
          		 Type inconsistency of struct and map when accessing the nested column Sub-task Resolved Ruifeng Zheng
          45.
          		 df["bad_key"] does not raise AnalysisException Sub-task Resolved Unassigned

          Activity

            People

              podongfeng Ruifeng Zheng
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: