[SPARK-38075] Hive script transform with order by and limit will return fake rows - ASF JIRA

XML

Word

Printable

JSON

For example:

create or replace temp view t as
select * from values
(1),
(2),
(3)
as t(a);

select transform(a)
USING 'cat' AS (a int)
FROM t order by a limit 10;

This returns:

NULL
NULL
NULL
1
2
3

Without order by and limit, the query returns:

1
2
3

Spark script transform does not have this issue. That is, if spark.sql.catalogImplementation=in-memory, Spark does not return fake rows.

links to

[Github] Pull Request #35368 (bersprockets)