Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2415

Samza-Sql: Fix AvroRelConverter to only consider cached schema while populating SamzaSqlRelRecord for all the nested records.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4
    • Component/s: None
    • Labels:
      None

      Description

      Currently, relRecord is populated with cached schema only at the top-level fields while the nested fields are populated with the schema of the incoming record. This ticket is to fix that inconsistency and always consider cached schema (schema that is fetched when the job starts).

      As a by-product of considering only cached schema, Sql job will keep running even though the input schema has evolved, let's say a new field is added. We will keep dropping the new field. As a separate effort, we need to consider adding metrics to flag whenever we detect that the input schema has evolved.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                atoomula Aditya Toomula
                Reporter:
                atoomula Aditya Toomula
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m