[IGNITE-12519] [Extensions] Spark SQL not working with NON upper case column names

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.6
• Component/s: spark
• Labels: dataframe spark spark-shell

Environment

1) Spark 2.3.0 (Tried on Mesos Master and Local Master)

2) Ignite 2.7.6 (10 Nodes Cluster on Kubernetes)

3) Spark Ignite 2.7.6

• Epic Link: spark-3.2
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Ivan Gagarkin
• Reporter: Praneeth Ramesh
• Reviewer: Andrei Aleksandrov
• Created: 01/Jan/20 00:27
• Updated: 05/Oct/22 20:02
• Resolved: 04/Oct/22 15:59

Description

I created a simple table as below.

CREATE TABLE acc (
  "accId" VARCHAR PRIMARY KEY,
  "accCol1" VARCHAR,
  "accCol2" INT,
  "accCol3" VARCHAR,
  "accCol4" BOOLEAN
);

And trying to read the data from table from Ignite Spark as below.

 

val igniteDF = spark.read
 .format(FORMAT_IGNITE)
 .option(OPTION_TABLE, "acc")
 .option(OPTION_CONFIG_FILE, "example-config.xml")
 .load()
igniteDF.show(100, false)

 

But I see an exception as below.

Caused by: org.h2.jdbc.JdbcSQLException: Column "ACCCOL1" not found; SQL statement:
SELECT accCol4, CAST(accCol1 AS VARCHAR) AS accCol1, accCol2, CAST(accCol3 AS VARCHAR) AS accCol3, accId FROM ACC LIMIT 21 [42122-197]
 at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
 at org.h2.message.DbException.get(DbException.java:179)
 at org.h2.message.DbException.get(DbException.java:155)
 at org.h2.expression.ExpressionColumn.optimize(ExpressionColumn.java:150)
 at org.h2.command.dml.Select.prepare(Select.java:858)
 at org.h2.command.Parser.prepareCommand(Parser.java:283)
 at org.h2.engine.Session.prepareLocal(Session.java:611)
 at org.h2.engine.Session.prepareCommand(Session.java:549)
 at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1247)
 at org.h2.jdbc.JdbcPreparedStatement.<init>(JdbcPreparedStatement.java:76)
 at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:694)
 at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.prepare0(IgniteH2Indexing.java:539)
 at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.prepareStatement(IgniteH2Indexing.java:509)
 at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.prepareStatement(IgniteH2Indexing.java:476)

 

When I try naming the TABLE cols with UPPER CASE everything works fine. But when I use the quotes in the Column Names to preserve the case, then it breaks with the exception.

From exception I can see query built is still having the UPPER case column name ACCCOL1 instead of the camel case column names.

Is there any workaround for this.

  

Issue Links

• links to: GitHub Pull Request #178