Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6418

Handle Schema change in Unnest And Lateral for unnest field / non-unnest field

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.14.0
    • Component/s: None
    • Labels:

      Description

      Handling the schema change scenarios for LATERAL and UNNEST when schema change is observed for non-unnest field and unnest field. It should also handle the scenario when UNNEST field is Repeated Map Type and schema change happens only in the children field of Map component. Following issues were found:

      1) There were issues found in how scan treats that kind of data where it scans 2 files, one has Map (let say cutomer_order) children type (custKey) as integer and other has custKey as string type. Then Scan just replaces the ValueVector from its output container for custKey but in the schema it has 2 fields with same name but different types.

      2) Unnest and Lateral check for schema change across batches based on the MaterializedField and BatchSchema reference they store. But since it's a reference any change in pointed value changes their reference as well and hence comparison always returns true. So for Unnest it actually keeps a clone of the Materialized Field instead of reference and use that to determine for schema change. For LATERAL any OK_NEW_SCHEMA from upstream is treated as actual schema change and setup is done again.

      3) Since the MaterializedField is mutable the isEquivalent method checks for object equality which is removed as that is not correct for mutable objects.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shamirwasia Sorabh Hamirwasia
                Reporter:
                shamirwasia Sorabh Hamirwasia
                Reviewer:
                Parth Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: