Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-12519

Spark SQL not working with NON upper case column names

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7.6
    • Fix Version/s: None
    • Component/s: spark
    • Environment:

      1) Spark 2.3.0 (Tried on Mesos Master and Local Master)

      2) Ignite 2.7.6 (10 Nodes Cluster on Kubernetes)

      3) Spark Ignite 2.7.6

    • Ignite Flags:
      Docs Required, Release Notes Required

      Description

      I created a simple table as below.

      CREATE TABLE acc (
        "accId" VARCHAR PRIMARY KEY,
        "accCol1" VARCHAR,
        "accCol2" INT,
        "accCol3" VARCHAR,
        "accCol4" BOOLEAN
      );

      And trying to read the data from table from Ignite Spark as below.

       

      val igniteDF = spark.read
       .format(FORMAT_IGNITE)
       .option(OPTION_TABLE, "acc")
       .option(OPTION_CONFIG_FILE, "example-config.xml")
       .load()
      igniteDF.show(100, false)
      

       

      But I see an exception as below.

      Caused by: org.h2.jdbc.JdbcSQLException: Column "ACCCOL1" not found; SQL statement:
      SELECT accCol4, CAST(accCol1 AS VARCHAR) AS accCol1, accCol2, CAST(accCol3 AS VARCHAR) AS accCol3, accId FROM ACC LIMIT 21 [42122-197]
       at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
       at org.h2.message.DbException.get(DbException.java:179)
       at org.h2.message.DbException.get(DbException.java:155)
       at org.h2.expression.ExpressionColumn.optimize(ExpressionColumn.java:150)
       at org.h2.command.dml.Select.prepare(Select.java:858)
       at org.h2.command.Parser.prepareCommand(Parser.java:283)
       at org.h2.engine.Session.prepareLocal(Session.java:611)
       at org.h2.engine.Session.prepareCommand(Session.java:549)
       at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1247)
       at org.h2.jdbc.JdbcPreparedStatement.<init>(JdbcPreparedStatement.java:76)
       at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:694)
       at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.prepare0(IgniteH2Indexing.java:539)
       at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.prepareStatement(IgniteH2Indexing.java:509)
       at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.prepareStatement(IgniteH2Indexing.java:476)

       

      When I try naming the TABLE cols with UPPER CASE everything works fine. But when I use the quotes in the Column Names to preserve the case, then it breaks with the exception.

      From exception I can see query built is still having the UPPER case column name ACCCOL1 instead of the camel case column names.

      Is there any workaround for this.

        

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sr.praneeth@gmail.com Praneeth Ramesh
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: