[PHOENIX-6783] Add spark-sql support for spark2 datasourceV2 and spark3

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: connectors-6.0.0
• Component/s: connectors, spark-connector

Environment

1. Pure Spark 3.0.3
2. phoenix5-spark3-shaded-6.0.0-SNAPSHOT.jar located in $SPARK_HOME/jars
3. SQL used to create Spark Phoenix data source table is:

create table Table1 (ID long, COL1 string) using  phoenix options (table 'Table1', zkUrl '192.168.0.103:2181', primary 'ID')

• Assignee: rejeb ben rejeb
• Reporter: yj
• Created: 02/Sep/22 08:01
• Updated: 18/Sep/24 20:36
• Resolved: 18/Sep/24 06:08

Description

While conducting the Spark 3.0 Integration test using Phoenix5-spark3-shaded module of project Phoenix-Connectors the following questions occurred.

 
The test process is as follows:

1. Phoenix created a table called "TABLE1" and then loaded the data into that table.

2. After that, I tried to select the data of the table using Spark.

 

As written in the README of Phoenix5-spark3 module, the part that loads Phoenix data into the Spark DataFrame using the Phoenix data source seems to work well.

However, when I create Phoenix Spark data source table through Spark SQL and try to select that data source table, the following error occurs:

java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: phoenix is not a valid Spark SQL Data Source

 

I wonder if the Phoenix-Connectors does not support creating Phoenix Spark data source tables, or if there is any other reason for this error.

Issue Links

• links to: GitHub Pull Request #139