[SPARK-3559] appendReadColumnIDs and appendReadColumnNames introduce unnecessary columns in the lists of needed column ids and column names stored in hiveConf

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: SQL
• Target Version/s: 1.2.0
• Assignee: Venkata Gollamudi
• Reporter: Yin Huai
• Created: 16/Sep/14 23:21
• Updated: 13/Oct/14 20:45
• Resolved: 13/Oct/14 20:45

Description

Because we are using the same hiveConf and we are currently using ColumnProjectionUtils.appendReadColumnIDs ColumnProjectionUtils.appendReadColumnNames to append needed column ids and names for a table, lists of needed column ids and names can have unnecessary columns.

Also, for a join operation, TableScanOperators for both tables are sharing the same hiveConf and they may need to set table-specific properties.

Issue Links

• is duplicated by: SPARK-3823 Spark Hive SQL readColumn is not reset each time for a new query
• links to: [Github] Pull Request #2713 (gvramana)