[SAMZA-2415] Samza-Sql: Fix AvroRelConverter to only consider cached schema while populating SamzaSqlRelRecord for all the nested records.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4
• Assignee: Aditya Toomula
• Reporter: Aditya Toomula
• Created: 09/Dec/19 17:58
• Updated: 01/Mar/20 16:28
• Resolved: 01/Mar/20 16:28

Description

Currently, relRecord is populated with cached schema only at the top-level fields while the nested fields are populated with the schema of the incoming record. This ticket is to fix that inconsistency and always consider cached schema (schema that is fetched when the job starts).

As a by-product of considering only cached schema, Sql job will keep running even though the input schema has evolved, let's say a new field is added. We will keep dropping the new field. As a separate effort, we need to consider adding metrics to flag whenever we detect that the input schema has evolved.

Issue Links

• links to: GitHub Pull Request #1233