[NIFI-8609] Improve efficiency of converting Record object to Avro GenericRecord object

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: Extensions
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 17/May/21 21:30
• Updated: 26/Oct/21 14:08
• Resolved: 15/Jun/21 14:06

Description

During some performance tests and profiling, I found that for a given flow, pushing Avro records to Kafka, one of the most expensive parts of the flow was converting our Record (MapRecord) object into a GenericRecord object for the Avro Writer.

I created a simple unit test to determine a baseline for performance numbers before making any changes. The unit test creates a Record with 100 null String fields, half of which have a null value assigned to them. I then converted the record into an Avro GenericRecord via AvroTypeUtil.createAvroRecord(record, avroSchema); in a loop of 1,000,000 iterations and output how long it took; this was then repeated 1,000 times in order to allow the JVM to warm up.

Numbers on my Macbook Pro showed after the first few iterations that the amount of time needed to convert 1 million records was on the order of 4.5 seconds.

After updating the code, performance numbers are just under 2 seconds. So somewhere on the order of 2x better performance.

Issue Links

• causes: NIFI-9335 When converting a NiFi Record to an Avro Record, default values in schema are not applied
• links to: GitHub Pull Request #5080