[HIVE-26507] Do not allow hive to iceberg migration if source table contains CHAR or VARCHAR columns

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-2
• Component/s: Iceberg integration
• Labels: iceberg pull-request-available
• Assignee: László Pintér
• Reporter: Rajesh Balamohan
• Created: 31/Aug/22 09:34
• Updated: 27/Jan/23 14:31
• Resolved: 07/Sep/22 11:00

Description

"alter table" statements can be used for generating iceberg metadata information (i.e for converting external tables  -> iceberg tables).

As part of this process, it also converts certain datatypes to iceberg compatible types (e.g char -> string). "iceberg.mr.schema.auto.conversion" enables this conversion.

This could cause certain issues at runtime. Here is an example

Before conversion:
==================
-- external table
select count(*) from customer_demographics where cd_gender = 'F' and cd_marital_status = 'U' and cd_education_status = '2 yr Degree';

27440

after conversion:
=================
-- iceberg table
select count(*) from customer_demographics where cd_gender = 'F' and cd_marital_status = 'U' and cd_education_status = '2 yr Degree';

0

select count(*) from customer_demographics where cd_gender = 'F' and cd_marital_status = 'U' and trim(cd_education_status) = '2 yr Degree';

27440
 

 

Issue Links

• relates to: HIVE-26980 CTAS and CMV fails if target table is Iceberg and source table has unsupported column type
• links to: GitHub Pull Request #3570
• links to: GitHub Pull Request #3593