[SPARK-36588] Use v2 commands by default

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: SQL
• Assignee: Terry Kim
• Reporter: Wenchen Fan
• Created: 25/Aug/21 14:16
• Updated: 06/May/22 10:12

Description

It's been a while after we introduce the v2 commands, and I think it's time to use v2 commands by default even for the session catalog, with a legacy config to fall back to the v1 commands.

We can do this one command by one command, with tests for both the v1 and v2 versions. The tests should help us understand the behavior difference between v1 and v2 commands, so that we can:

1. fix the v2 commands to match the v1 behavior
2. or accept the behavior difference and write migration guide

We can reuse the test framework built in https://issues.apache.org/jira/browse/SPARK-33381

Sub-Tasks

• Migrate SHOW CREATE TABLE to use V2 command by default: Open
• Keep consistent order of columns with user specify for v1 table: In Progress
• Migrate SHOW TABLES EXTENDED to use v2 command by default: Open
• Migrate DESCRIBE TABLE to use V2 command by default: Open