Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Currently there are several operators such as sort, group by and join that assume a constant schema. One example of a bug introduced by this assumption is the current functionality of breaking a group by grouping any time we hit a schema change in the batch, even if it is not in the column being grouped by. This is a limitation of the constant schema in a batch, even if we have all integers in one column we are grouping by, we cannot hold all of the records matching a group by constraint in a single batch if the type changes in another column. This currently forces a breaking up of the records that should not happen. There are also cases where the column we are interested in for the group by, sort, etc. should be able to incorporate schema changes into the algorithm, such as correctly sorting different numeric types together or doing a similar aggregation with compatible types, such as the different numeric types.