For now, persistent Default Region makes System cache persistent as well (same correct for non-persistent region). This behavior is non-obvious and it may causes unpredictable issues.
We have number of components that uses system cache, some of them doesn't need system cache to be persistent, while other ok with it:
- DataStructures - datastructures persistence should be configured in it's configuration. Moreover, some structures looks broken as CacheDataStructureManages uses in-memory maps.
- Compute - most likely persistence not needed.
- Services - metastore can be used instead.
- Igfs - candidate to remove in 3.0
Let's make it non-persistent, fix components that require persistence and may be deprecate system cache.