Major Description
1. It is impossible to distinguish empty string and null, for instance:
scala> Seq(Seq(""), Seq(null)).toDF().show
+-----+
|value|
+-----+
| []|
| []|
+-----+
2. Inconsistent NULL conversions for top-level values and nested columns, for instance:
scala> sql("select named_struct('c', null), null").show +---------------------+----+ |named_struct(c, NULL)|NULL| +---------------------+----+ | []|null| +---------------------+----+
3. `.show()` is different from conversions to Hive strings, and as a consequence its output is different from `spark-sql` (sql tests):
spark-sql> select named_struct('c', null) as struct; {"c":null}
scala> sql("select named_struct('c', null) as struct").show
+------+
|struct|
+------+
| []|
+------+
Attachments
Issue Links
- is related to
-
SPARK-33291 Inconsistent NULL conversions to strings redux
-
- Resolved
-
- links to
