Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Handling the schema change scenarios for LATERAL and UNNEST when schema change is observed for non-unnest field and unnest field. It should also handle the scenario when UNNEST field is Repeated Map Type and schema change happens only in the children field of Map component. Following issues were found:
1) There were issues found in how scan treats that kind of data where it scans 2 files, one has Map (let say cutomer_order) children type (custKey) as integer and other has custKey as string type. Then Scan just replaces the ValueVector from its output container for custKey but in the schema it has 2 fields with same name but different types.
2) Unnest and Lateral check for schema change across batches based on the MaterializedField and BatchSchema reference they store. But since it's a reference any change in pointed value changes their reference as well and hence comparison always returns true. So for Unnest it actually keeps a clone of the Materialized Field instead of reference and use that to determine for schema change. For LATERAL any OK_NEW_SCHEMA from upstream is treated as actual schema change and setup is done again.
3) Since the MaterializedField is mutable the isEquivalent method checks for object equality which is removed as that is not correct for mutable objects.
Attachments
Issue Links
- is depended upon by
-
DRILL-5999 Add support for LATERAL join
-
- In Progress
-
- links to