[SPARK-27726] Performance of InMemoryStore suffers under load

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.3
• Fix Version/s: 2.4.4, 3.0.0
• Component/s: Spark Core
• Assignee: David C Navas
• Reporter: David C Navas
• Created: 15/May/19 18:12
• Updated: 21/May/19 18:14
• Resolved: 21/May/19 17:23

Description

When our Spark system has been under load for an extended period of time, GC remains highly active and the jobs page becomes unresponsive even when load eases.  Please see the attached GCRateIssues for more details regarding the problem definition.

We found a number of separate issues which are detailed in the subtasks.  I anticipate committing a single PR for all subtasks whose commits roughly align with the descriptions in the subtasks.

The performance of the code is measured before and after the change and is attached in the document PerformanceBeforeAndAfter.  tl;dr in our use case, we saw about five (computed) orders of magnitude improvement.

Issue Links

• is duplicated by: SPARK-27727 Asynchronous ElementStore cleanup should have only one pending cleanup per class
• is duplicated by: SPARK-27728 Address thread-safety of InMemoryStore and ElementTrackingStores.
• is duplicated by: SPARK-27729 Extract deletion of the summaries from the stage deletion loop
• is duplicated by: SPARK-27730 Add support for removeAllKeys
• is duplicated by: SPARK-27731 Cleanup some non-compile time type checking and exception handling
• links to: GitHub Pull Request #24616

Sub-Tasks

• Asynchronous ElementStore cleanup should have only one pending cleanup per class: Closed
• Address thread-safety of InMemoryStore and ElementTrackingStores.: Closed
• Extract deletion of the summaries from the stage deletion loop: Closed
• Add support for removeAllKeys: Closed
• Cleanup some non-compile time type checking and exception handling: Closed