[AMBARI-25572] Metrics cannot be stored if mutation size is not set properly

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.3, 2.7.4, 2.7.5
• Component/s: ambari-metrics
• Labels: metric-collector reliability
• Assignee: Unassigned
• Reporter: Tamas Payer
• Created: 22/Oct/20 10:14
• Updated: 16/Jun/21 08:39

Description

Ambari Metrics Collector sometimes fails to store metric because the batch size settings does not fit to the current row size and the assembled batch probably does not fit into the maximum batch memory space.

2020-10-20 10:29:04,752 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Failed on insert records to store : ERROR 730 (LIM02): MutationState size is bigger than maximum allowed number of bytes
2020-10-20 10:29:04,752 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Metric that cannot be stored : [kafka.server.ConsumerGroupMetrics.CommittedOffset.clientId.-.group.console-consumer-46399.partition.0.topic.PREPROD_KAFKA_CBS_TRANSACTION._sum,kafka_broker]{1603186125049=2.65274693E8}

This error can be eliminated by the tweaking of phoenix.mutate.batchSize and phoenix.mutate.batchSizeBytes configuration settings.

Asses the feasibility of changing the AMS implementation that retries upon failure after manipulation the auto-commit or batch size, or whatever is needed for the successful commit and notifies the user about sub optimal batch size settings.