[NIFI-8167] Decouple NiFi & Schema Registry dependencies

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Core Framework
• Assignee: Unassigned
• Reporter: Rahul Soni
• Created: 22/Jan/21 15:32
• Updated: 22/Jan/21 15:32

Description

Today, NiFi depends on Schema Registry for Avro schemas for the data it is processing. By default, the caching duration for the Schema Registry controller service is 1 hour, which can be of course increased or decreased. But it has its flaws.

• Make the cache duration too high, and you would need to wait for a long time before an updated schema is reflected in the NiFi flow.
• Make the cache duration too low, and the NiFi flow may spend higher time in fetching the schemas from SR and also may fail more often with schema not available errors, in case the Schema Registry instance is not available for a short period.

I would suggest having Schema Registry behave like Ranger do. All the schemas from the Schema Registry can be cached at the NiFi nodes like Ranger's policy cache and any change to the schema can be propagated to the NiFi nodes using that Schema Registry instance.