[SPARK-26699] Dataset column output discrepancies

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.3.2
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Praveena
• Created: 23/Jan/19 07:13
• Updated: 30/Jan/19 06:57
• Resolved: 24/Jan/19 06:24

Description

Hi,

 

When i run my job in Local mode (meaning as standalone in Eclipse) with same parquet input files, the output is -

 

locations
--------------------
[[[true, [[, phys...
[[[true, [[, phys...
[[[true, [[, phys...
null
[[[true, [[, phys...
[[[true, [[, phys...
[[[true, [[, phys...
[[[true, [[, phys...
[[[true, [[, phys...
[[[true, [[, phys...

 

But when i run the same code base with same input parquet files in the YARN cluster mode, my output is as below -

--------------------
locations
--------------------
[WrappedArray([tr...
[WrappedArray([tr...
[WrappedArray([tr...
null
[WrappedArray([tr...
[WrappedArray([tr...
[WrappedArray([tr...
[WrappedArray([tr...
[WrappedArray([tr...
[WrappedArray([tr...

Its appending WrappedArray

I am using Apache Spark 2.3.2 version and the EMR Version is 5.19.0. What could be the reason for discrepancies in the output of certain Table columns ?