[SPARK-41790] Set TRANSFORM reader and writer's format correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: mattshma
• Reporter: mattshma
• Created: 30/Dec/22 13:12
• Updated: 05/Jan/23 08:53
• Resolved: 05/Jan/23 08:53

Description

We'll get wrong data when transform only specify reader or writer 's row format delimited, the reason is using the wrong format to feed/fetch data to/from running script now.  In theory, writer uses inFormat to feed to input data into the running script and reader uses outFormat to read the output from the running script, but inFormat and outFormat are set wrong value currently in the following code:

val (inFormat, inSerdeClass, inSerdeProps, reader) =
  format(
    inRowFormat, "hive.script.recordreader",
    "org.apache.hadoop.hive.ql.exec.TextRecordReader")

val (outFormat, outSerdeClass, outSerdeProps, writer) =
  format(
    outRowFormat, "hive.script.recordwriter",
    "org.apache.hadoop.hive.ql.exec.TextRecordWriter") 

 

Example SQL:

spark-sql> CREATE TABLE t1 (a string, b string); 

spark-sql> INSERT OVERWRITE t1 VALUES("1", "2"), ("3", "4");

spark-sql> SELECT TRANSFORM(a, b)
         >   ROW FORMAT DELIMITED
         >   FIELDS TERMINATED BY ','
         >   USING 'cat'
         >   AS (c)
         > FROM t1;
c

spark-sql> SELECT TRANSFORM(a, b)
         >   USING 'cat'
         >   AS (c)
         >   ROW FORMAT DELIMITED
         >   FIELDS TERMINATED BY ','
         > FROM t1;
c
1    23    4

 

The same sql in hive:

hive> SELECT TRANSFORM(a, b)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY ','
    >   USING 'cat'
    >   AS (c)
    > FROM t1;
c
1,2
3,4

hive> SELECT TRANSFORM(a, b)
    >   USING 'cat'
    >   AS (c)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY ','
    > FROM t1;
c
1    2
3    4 

 

Issue Links

• links to: [Github] Pull Request #39315 (mattshma)
• links to: [Github] Pull Request #39315 (mattshma)