Details
-
Improvement
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Since there is no schema validation for external tables, users may face various errors if their orc data and external table schema does not match. If orc schema has fewer columns than projection OrcEncodedDataConsumer may receive an incomplete TypeDescription array which will manifest itself as NullPointerException later.
We can at least verify that OrcEncodedDataConsumer gets enough TypeDescriptions. If assertion fails, user sees there is something wrong with the schema and hopefully resolves the problem quickly. If there are enough columns in the file but the schema of the query does not match, user generally sees a ClassCastException. If there are enough columns and types accidentally match, there is nothing we can do as this is an external table.
We have seen this when trying to use a managed table as external table location. Although user facing schemas are the same, managed table has acid related metadata. I am adding a q file demonstrating NullPointerException with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added this to precommit tests as it is hard to assert the exception message from mini driver framework and effectively it is just changing the error.