[IGNITE-12183] Rebalancing process metrics for cache groups

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: IEP-35
• Ignite Flags: Docs Required
• Assignee: Unassigned
• Reporter: Aleksandr Brazhnikov
• Created: 18/Sep/19 08:48
• Updated: 20/Dec/19 11:28

Description

Intro

Currently, some of the Apache Ignite rebalance process metrics are not working well enough. For instance, `EstimatedRebalancingKeys` keys time to time returns `-1` value due to a bug, or `rebalanceKeysReceived` metric treated as CacheMetric in fact calculated for the whole cache group (e.g. historical rebalance, see IGNITE-11330 and code block comment below).

All the rebalance process metrics must be re-worked.

    /**
     * Update rebalancing metrics.
     */
    private void updateGroupMetrics() {
        // TODO: IGNITE-11330: Update metrics for touched cache only.
        // Due to historical rebalancing "EstimatedRebalancingKeys" metric is currently calculated for the whole cache
        // group (by partition counters), so "RebalancedKeys" and "RebalancingKeysRate" is calculated in the same way.
        for (GridCacheContext cctx0 : grp.caches()) {
            if (cctx0.statisticsEnabled())
                cctx0.cache().metrics0().onRebalanceKeyReceived();
        }
    }

What we have

CacheMetrics - statistics must be enabled to see these metrics.

• getRebalancedKeys
• getKeysToRebalanceLeft
• getEstimatedRebalancingKeys
• getEstimatedRebalancingFinishTime
• getRebalancingStartTime
• getRebalanceClearingPartitionsLeft
• getRebalancingKeysRate
• getRebalancingBytesRate

What to do

All such metrics (or their analogue) must be available for the CacheGroupMetrics. I'd suggest to do the following:

1. Phase-1
   • rebalancingPartitionsLeft long metric
   • rebalancingReceivedKeys long metric
   • rebalancingReceivedBytes long metric
   • rebalancingStartTime long metric
   • rebalancingFinishTime long metric
   • rebalancingLastCancelledTime long metric
2. Phase-2
   • rebalancingExpectedKeys long metric
   • rebalancingExpectedBytes long metric
   • rebalancingEvictedPartitionsLeft long metric
3. Phase-3 (statistics must be enabled)
   • rebalancingKeysRate HitRate metric
   • rebalancingBytesRate HitRate metric
4. Phase-4
   • Mark rebalancing CacheMetrics deprecated and remove from metrics framework IGNITE-11961.

Issue Links

• is related to: IGNITE-11330 Partition demander should skip entries for invalid partition.
• links to: Apache Ignite dev-list discussion
• links to: GitHub Pull Request #6959

Sub-Tasks

• [Phase-2] Calculate expected rebalancing cache group keys: Patch Available
• [Phase-3] Rebalance HitRate metrics: Open
• [Phase-4] Deprecate old rebalancing cache metrics: Open