[HIVE-25429] Delta metrics collection may cause number of tez counters to exceed tez.counters.max limit

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Hive
• Labels: pull-request-available
• Assignee: Karen Coppage
• Reporter: Karen Coppage
• Created: 05/Aug/21 10:09
• Updated: 17/Nov/22 08:48
• Resolved: 27/Aug/21 09:08

Description

There's a limit to the number of tez counters allowed (tez.counters.max). Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 counters for each partition touched by a given query, which can result in a huge number of counters, which is unnecessary because we're only interested in n number of partitions with the most deltas. This change limits the number of counters created to hive.txn.acid.metrics.max.cache.size*3.

Also when tez.counters.max is reached a LimitExceededException is thrown but isn't caught on the Hive side and causes the query to fail. We should catch this and skip delta metrics collection in this case.

Also make sure that metrics are only collected if hive.metastore.acidmetrics.ext.on=true

Issue Links

• links to: GitHub Pull Request #2563