[SPARK-22805] Use aliases for StorageLevel in event logs

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.1.2, 2.2.1
• Component/s: Spark Core
• Assignee: Unassigned Assign to me
• Reporter: Sergei Lebedev
• Created: 15/Dec/17 16:15
• Updated: 21/Jan/18 00:24
• Resolved: 21/Jan/18 00:24

Description

Fact 1: StorageLevel has a private constructor, therefore a list of predefined levels is not extendable (by the users).

Fact 2: The format of event logs uses redundant representation for storage levels

>>> len('{"Use Disk": true, "Use Memory": false, "Deserialized": true, "Replication": 1}')
79
>>> len('DISK_ONLY')
9

Fact 3: This leads to excessive log sizes for workloads with lots of partitions, because every partition would have the storage level field which is 60-70 bytes more than it should be.

Suggested quick win: use the names of the predefined levels to identify them in the event log.

Issue Links

• links to: [Github] Pull Request #19992 (superbobry)