[FLINK-16839] Connectors Kinesis metrics can be disabled

• Type: Improvement
• Status: Reopened
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.10.0
• Component/s: Connectors / Kinesis, Runtime / Metrics
• Assignee: Unassigned
• Reporter: Muchl
• Created: 28/Mar/20 05:29
• Updated: 29/Jul/21 07:38

Description

Currently there are 9 metrics in the kinesis connector, each of which is recorded according to the kinesis shard dimension. If there are enough shards, taskmanager mtrics will be unavailable.

In our production environment, a case of a job that reads dynamodb stream kinesis adapter, this dynamodb has more than 10,000 shards, multiplied by 9 metrics, there are more than 100,000 metrics for kinesis, and the entire metrics output reaches tens of MB , Cause prometheus cannot collect metrics.

We should have a configuration item that can disable kinesis metrics