[HUDI-512] Support Logical Partitioning with Expression Index

• Type: Epic
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 1.0.0
• Component/s: Common Core
• Labels: features pull-request-available
• Epic Name: Indexing Functions
• Assignee: Ethan Guo (this is the old account; please use "yihua")
• Reporter: Alexander Filipchik
• Created: 08/Jan/20 05:32
• Updated: 13/Dec/24 01:03
• Resolved: 11/Dec/24 23:39

Description

So for us to support logical partitioning in lieu of physical one following will be necessary: 

1. If User would like to apply any transformations on top of raw Partitioning column, such transformed column will have to be materialized (either in the table as meta-column or elsewhere)
2. We will have to make sure that individual Base (alas Delta Log) files only contain records with the same partition values (ie records have to be implicitly clustered by partition values w/in files)
3. Partition Values have to be exposed to the Query Engine such that Partition pruning could be performed (limiting number of files that will be scanned) 

 

---

--- Original Description ---

This one is more inspirational, but, I believe, will be very useful. Currently hudi is following Hive table format, which means that data is logically and physically partitioned into folder structure like:

table_name

  2019

    01

    02

       bla.parquet

 

This has several issues:

 1) Modern object stores (AWS S3, GCP) are more performant when each file name starts with some kind of a random value. By definition Hive layout is not perfect

2) Hive Metastore stores partitions in the text field in the single table (2 tables with very similar information) and doesn't support proper filtering. Data partitioned by day will be stored like:

2019/01/10

2019/01/11

so only regexp queries are suported (at least in Hive 2.X.X)

3) Having a single POF which relies on non distributed DB is dangerous and creates bottlenecks. 

 

The idea is to get rid of logical partitioning all together (and hive metastore as well). If dataset has a time columns, user should be able to query it without understanding what is the physical layout of the table (by specifying those partitions explicitly or ending up with a full table scan accidentally).

It will require some kind of mapping of time to file locations (similar to Iceberg). I'm also leaning towards the idea that storing table metadata with the table is a good thing as it can be read by the engine in one shot and will be faster that taxing a standalone metastore. 

Issue Links

• is related to: HUDI-3594 Support standard Spark functions in Filter Exprs in Data Skipping