Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.12.0
-
None
-
None
-
None
Description
Currently there is a time between generating/deserializing a scan token and opening the scanner that can result in and invalid schema when scanning the table. This is especially the case when a column is renamed or dropped and then another column with the same name is created.
This has been somewhat worked around client side by leveraging column ids and mapping the scan token projection to the new schema based on the column ids. However, this doesn't work when the scan token sends it's own metadata (KUDU-1802).
We should provide a mechanism to allow the schema to be consistent and guaranteed to work from the point a scan token is generated to the time it is run/completed.
A simple approach might be to allow column ids to be passed on the scan request. Instead of handling the mapping client side, this makes the column ids more of a server side concern again (which was the original intent). This is also how table renames are handled, by passing the table id. Of course this wouldn't support destructive changes such as dropping a column, but that would require a much larger change to keep the dropped column for a period of time and use the snapshot time to scan using the schema at the given snapshot time.
The other open issue with column name changes in todays implementation is that predicates are by name only and the client can't map them the same way it does for the projection. Using column ids in the predicates would be required as well if that approach is taken.