[IGNITE-10314] Spark dataframe will get wrong schema if user executes add/drop column DDL

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3, 2.4, 2.5, 2.6, 2.7
• Fix Version/s: 2.8
• Component/s: spark
• Assignee: Ray Liu
• Reporter: Ray Liu
• Created: 19/Nov/18 03:47
• Updated: 10/Oct/19 14:42
• Resolved: 21/Jan/19 08:37

Description

When user performs add/remove column in DDL,  Spark will get the old/wrong schema.

 

Analyse 

Currently Spark data frame API relies on QueryEntity to construct schema, but QueryEntity in QuerySchema is a local copy of the original QueryEntity, so the original QueryEntity is not updated when modification happens.

 

Solution

Use GridQueryTypeDescriptor to replace QueryEntity

Issue Links

• is blocked by: IGNITE-10356 JDBC thin driver returns wrong data type for Date and Decimal SQL type
• is duplicated by: IGNITE-12159 Ignite spark doesn't support Alter Column syntax
• links to: GitHub Pull Request #5598