[YUNIKORN-1483] Overhaul periodic state dump support

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: core - scheduler, release, shim - kubernetes
• Assignee: Craig Condit
• Reporter: Craig Condit
• Created: 19/Dec/22 10:10
• Updated: 22/Dec/22 15:31
• Resolved: 22/Dec/22 15:31

Description

The current support for generating periodic state dumps implemented in YUNIKORN-940 has several warts:

1. The configuration in YUNIKORN-949 is done via the core scheduler configuration, leading to a random option on partitions which doesn't belong there and has nothing to do with scheduling.
2. Changing the frequency of the state dumps is done via the unsecured REST API. This is a potential denial-of-service vector.
3. Configuration V2 is now complete, which standardizes on using a ConfigMap to configure all YuniKorn options that make sense to be reconfigured. However, allowing the location to be changed at runtime makes no sense in a containerized environment.
4. Retrieving the state dumps requires mounting of external storage. This is necessarily a site-specific configuration and currently requires a custom Helm deployment.
5. The state dumps, though JSON, are emitted as text files with JSON appended to them, making parsing difficult.

To address these issues:

1. Deprecate existing REST API configuration for frequency, and make it a no-op now for security reasons. We can remove it completely in 2.0.
2. Deprecate the statedumpfilepath option on partitions. Ignore it for security reasons now (and warn if found), and remove completely in 2.0.
3. Disable the feature by default. To enable it, we should require setting a specific environment variable:
   • YUNIKORN_STATE_DUMP_LOCATION=/path/to/dir : This would be required to enable the feature at all. Making it an env var makes sense as it is not an option that should be reconfigured (or even visible) in configuration.

1. Via configmap, we should allow the feature to be enabled / disabled and its frequency set. These options would have no effect if YUNIKORN_STATE_DUMP_LOCATION is not defined:
   • periodicStateDump.enabled: "true" | "false" (default "false")
   • periodicStateDump.frequency: "15m" (default value, do not allow more frequently than 1m intervals)
   • periodicStateDump.count: 10 (default value)
2. Create an empty directory /yunkorn-state in the Docker image to store state dumps.
3. Add support to Helm for enabling state dump support as well as setting custom mount options (including quota). Enabling support should set the env var YUNIKORN_STATE_DUMP_LOCATION=/yunikorn-state and mount this directory via the options specified.
4. Output a single json file per dump and remove oldest files until count <= periodicStateDump.count entries: yunikorn-state-dump-YYYYMMDD-HHMM.json

 

Issue Links

• is related to: YUNIKORN-940 Periodic & on-demand state dump in Yunikorn
• is related to: YUNIKORN-949 Location of the state dump file should be configurable
• is related to: YUNIKORN-1069 Make the state dump file's log rotation parameters configurable
• is superceded by: YUNIKORN-1500 Remove periodic state dump support