In an offline conversation with Jan Høydahl we identified the following areas of improvement to the metrics history API in order to increase its user-friendliness and provide more details about its status.
- there are three possible states for the API: inactive (when not in cloud mode), in-memory only (when .system collection doesn’t exist), and persisted when it’s both active and persisted in Solr. The /admin/metrics/history endpoint should give some hint about this status, such as "mode":"memory/index", "active": true|false. Or a separate action=status just to poll status? Currently when the API is inactive it simply returns 404 Not Found.
- when in "memory" mode a call to /admin/metrics/history on a non-overseer node should forward the request to the overseer, so that the client does not need to care what mode it is in - kind of like how a query works in distributed mode.
- better documentation for the API behavior in each mode.
- perhaps if mode=memory, there could also be a "message":"Warning, metrics history is not being persisted. Please create the .system collection to start persisting history"