[SPARK-19316] Spark event logs are huge compared to 1.5.2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0
• Assignee: Unassigned
• Reporter: Jisoo Kim
• Created: 20/Jan/17 21:44
• Updated: 26/Jan/17 20:06
• Resolved: 21/Jan/17 14:10

Description

I have a Spark application with many tasks (more than 40k). The event logs for such application used to be around 2g when I was using Spark 1.5.2 standalone cluster. Now that I am using Spark 2.0 with Mesos, the size of the event log of such application drastically increased from 2g to 60g with a similar number of tasks. This is affecting Spark History Server since it is having trouble reading such huge event log. I wonder the increase in a size of an event log is expected in Spark 2.0.

Issue Links

• duplicates: SPARK-16333 Excessive Spark history event/json data size (5GB each)