[SPARK-41794] Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_column

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Connect, Tests
• Labels: pull-request-available
• Assignee: Ruifeng Zheng
• Reporter: Hyukjin Kwon
• Created: 31/Dec/22 01:38
• Updated: 14/May/24 03:53

Description

======================================================================
ERROR [0.901s]: test_column_accessor (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../spark/python/pyspark/sql/tests/connect/test_connect_column.py", line 744, in test_column_accessor
    cdf.select(CF.col("z")[0], cdf.z[10], CF.col("z")[-10]).toPandas(),
  File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 949, in toPandas
    return self._session.client.to_pandas(query)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 413, in to_pandas
    return self._execute_and_fetch(req)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
    self._handle_error(rpc_error)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 623, in _handle_error
    raise SparkConnectException(status.message, info.reason) from None
pyspark.sql.connect.client.SparkConnectException: (org.apache.spark.SparkArrayIndexOutOfBoundsException) [INVALID_ARRAY_INDEX] The index 10 is out of bounds. The array has 3 elements. Use the SQL function `get()` to tolerate accessing element at invalid index and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

======================================================================
ERROR [0.245s]: test_column_arithmetic_ops (pyspark.sql.tests.connect.test_connect_column.SparkConnectTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../spark/python/pyspark/sql/tests/connect/test_connect_column.py", line 799, in test_column_arithmetic_ops
    cdf.select(cdf.a % cdf["b"], cdf["a"] % 2, 12 % cdf.c).toPandas(),
  File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 949, in toPandas
    return self._session.client.to_pandas(query)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 413, in to_pandas
    return self._execute_and_fetch(req)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
    self._handle_error(rpc_error)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 623, in _handle_error
    raise SparkConnectException(status.message, info.reason) from None
pyspark.sql.connect.client.SparkConnectException: (org.apache.spark.SparkArithmeticException) [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

Issue Links

• is related to: SPARK-41795 Disable ANSI mode in pyspark.sql.tests.connect.test_connect_column
• is related to: SPARK-44444 Use ANSI SQL mode by default
• links to: GitHub Pull Request #46434
• links to: GitHub Pull Request #46566