[SPARK-16716] calling cache on joined dataframe can lead to data being blanked

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.2
• Component/s: SQL
• Assignee: Unassigned
• Reporter: PJ Fanning
• Created: 25/Jul/16 20:37
• Updated: 13/Aug/16 02:09
• Resolved: 13/Aug/16 02:09

Description

I have reproduced the issue in Spark 1.6.2 and latest 1.6.3-SNAPSHOT code.
The code works ok on Spark 1.6.1.
I have a notebook up on Databricks Community Edition that demonstrates the issue. The notebook depends on the library com.databricks:spark-csv_2.10:1.4.0
The code uses some custom code to join 4 dataframes.
It calls show on this dataframe and the data is as expected.
After calling .cache, the data is blanked.

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/5458351705459939/3760010872339805/5521341683971298/latest.html

Issue Links

• duplicates: SPARK-16664 Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.