[SPARK-50092] Fix PostgreSQL connector behaviour for multidimensional arrays

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Petar Vasiljevic
• Reporter: Petar Vasiljevic
• Created: 23/Oct/24 17:40
• Updated: 14/Nov/24 03:51
• Resolved: 14/Nov/24 03:51

Description

There is a bug introduced in this PR https://github.com/apache/spark/pull/46006. This PR fixed the behaviour for PostgreSQL connector for multidimensional arrays since we have mapped all arrays to 1D arrays.

This PR has introduced a bug. Following scenario is broken:

• User has a table t1 on Postgres and does CTAS command to create table t2 with same data.
• PR 46006 is resolving the dimensionality of column by reading the metadata from pg_attribute table and attndims column.
• This query returns correct dimensionality for table t1, but for table t2 that is created via CTAS it returns 0 always. This leads to all of the arrays being mapped to 0-D array which is the type itself (for example int)

As a solution, we can query array_ndims function on given column that will return the dimension of the column. It works for CTAS-like-created tables too.

Issue Links

• links to: GitHub Pull Request #48625