Type: IT Help
Resolution: Not A Problem
Affects Version/s: 2.3.4
Fix Version/s: None
Component/s: Spark Submit
I am migrating from spark 1.6 to spark 2.3. However in collect_list I am getting different schema.
When I am running above line in spark 1.6 getting below schema
but in spark 2.3 schema changed to
In Spark 1.6 array($"b",$"c",$"data1") is converting to string like this
In spark 2.3 it is converted to WrappedArray
I want to keep my schema as it is Otherwise all the dependent codes have to change.