[IMPALA-10947] SQL support for querying Iceberg metadata

• Type: Epic
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Frontend
• Labels: impala-iceberg
• Target Version: Impala 4.3.0
• Epic Color: ghx-label-5
• Assignee: Daniel Becker
• Reporter: Zoltán Borók-Nagy
• Created: 01/Oct/21 13:28
• Updated: 16/May/24 14:12

Description

HIVE-25457 added support for querying Iceberg table metadata to Hive.

They support the following syntax:
SELECT * FROM default.iceberg_table.history;

Spark uses the same syntax: https://iceberg.apache.org/spark-queries/#history

Other than "history", the following metadata tables are available in Iceberg:
The following metadata tables are available in Iceberg:

• ENTRIES,
• FILES,
• HISTORY,
• SNAPSHOTS,
• MANIFESTS,
• PARTITIONS,
• ALL_DATA_FILES,
• ALL_MANIFESTS,
• ALL_ENTRIES

Impala currently only supports "DESCRIBE HISTORY <tbl>". The above SELECT syntax would be more convenient for the users, also it would be more flexible as users could easily define filters in WHERE clauses. And of course we would be consistent with other engines.

Issue Links

• causes: IMPALA-12661 ASAN heap-use-after-free in IcebergMetadataScanNode
• causes: IMPALA-12706 Failing DCHECK when querying STRUCT inside a STRUCT for Iceberg metadata table
• is related to: IMPALA-12853 Collection types in FROM clause for Iceberg Metadata Tables
• is related to: IMPALA-12809 Iceberg metadata table scanner should always be scheduled to the coordinator
• is related to: IMPALA-13035 Querying metadata tables from non-Iceberg tables throws IllegalArgumentException
• relates to: IMPALA-12845 Crash with DESCRIBE on a complex type from an Iceberg table
• relates to: IMPALA-13036 Document Iceberg metadata tables

Sub-Tasks

• Iceberg Metadata querying parser change: Resolved
• Iceberg Metadata querying planner change: Resolved
• Iceberg Metadata querying executor change: Resolved
• Add support to STRUCT type Iceberg Metadata table columns: Resolved
• Add support to ARRAY type Iceberg Metadata table columns: Resolved
• Add support to MAP type Iceberg Metadata table columns: Resolved
• Add support to BINARY type Iceberg Metadata table columns: Resolved
• Temporary workaround for BINARY in complex types: Resolved
• Enable EXPAND_COMPLEX_TYPES for Iceberg Metadata tables: Resolved
• Iceberg Metadata querying with time travel: Open
• test_metadata_tables could occasionally fail in the s3 build: Resolved
• Describe command for Iceberg metadata tables: Resolved
• Implement SHOW METADATA TABLES IN statement to list Iceberg Metadata tables: Resolved
• Add support for BINARY in complex types in Iceberg metadata tables: Resolved
• Add support for DATE in Iceberg metadata tables: Resolved
• Add support for FLOAT/DOUBLE in Iceberg metadata tables: Resolved
• Add support for DECIMAL in Iceberg metadata tables: Open