Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22273

Fix key/value schema field names in HashMapGenerators.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.1, 2.2.0
    • Fix Version/s: 2.1.3, 2.2.1, 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      When fixing schema field names using escape characters with addReferenceMinorObj() at SPARK-18952, double-quotes around the names were remained and the names become something like "((java.lang.String) references[1])".

      /* 055 */     private int maxSteps = 2;
      /* 056 */     private int numRows = 0;
      /* 057 */     private org.apache.spark.sql.types.StructType keySchema = new org.apache.spark.sql.types.StructType().add("((java.lang.String) references[1])", org.apache.spark.sql.types.DataTypes.StringType);
      /* 058 */     private org.apache.spark.sql.types.StructType valueSchema = new org.apache.spark.sql.types.StructType().add("((java.lang.String) references[2])", org.apache.spark.sql.types.DataTypes.LongType);
      /* 059 */     private Object emptyVBase;
      

      We should remove the double-quotes to refer the values in references properly:

      /* 055 */     private int maxSteps = 2;
      /* 056 */     private int numRows = 0;
      /* 057 */     private org.apache.spark.sql.types.StructType keySchema = new org.apache.spark.sql.types.StructType().add(((java.lang.String) references[1]), org.apache.spark.sql.types.DataTypes.StringType);
      /* 058 */     private org.apache.spark.sql.types.StructType valueSchema = new org.apache.spark.sql.types.StructType().add(((java.lang.String) references[2]), org.apache.spark.sql.types.DataTypes.LongType);
      /* 059 */     private Object emptyVBase;
      

        Attachments

          Activity

            People

            • Assignee:
              ueshin Takuya Ueshin
              Reporter:
              ueshin Takuya Ueshin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: