[PHOENIX-6667] Spark3 connector requires that all columns are specified when writing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: connectors-6.0.0
• Fix Version/s: connectors-6.0.0
• Component/s: connectors, spark-connector
• Assignee: Attila Zsolt Piros
• Reporter: Istvan Toth
• Created: 16/Mar/22 10:11
• Updated: 06/Apr/23 14:14
• Resolved: 06/Apr/23 14:12

Description

For Spark 2, it was possible to omit some columns from the dataframe, the same way it is not mandatory to specify all columns when upserting via SQL.

Spark3 has added new checks, which require that EVERY sql column is specifed in the DataFrame.

Consequently, when using the current API, writing will fail unless you specify all columns.

This is a loss of functionality WRT Phoenix (and other SQL datastores) compared to Spark2.

I don't think that we can do anything from the Phoenix side, just documenting the regression here.

Maybe future Spark versions will make this configurable.

Issue Links

• fixes: PHOENIX-6668 Spark3 connector cannot distinguish column name cases
• is broken by: PHOENIX-6632 Migrate connectors to Spark-3
• links to: GitHub Pull Request #95
• links to: GitHub Pull Request #96