[SPARK-20651] Speed up the new app state listener

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 2.3.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Marcelo Masiero Vanzin
• Created: 08/May/17 18:05
• Updated: 30/May/17 23:11
• Resolved: 30/May/17 23:11

Description

See spec in parent issue (SPARK-18085) for more details.

This task tracks enhancements to the code added in previous tasks so that the new app state listener is faster; it adds a caching layer and an asynchronous write layer that also does deduplication, so that it avoids blocking the listener thread and also avoids unnecessary writes to disk.