[KYLIN-5794] Add Spark parquet footer read cache - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Duplicate
Affects Version/s: None
Fix Version/s: None
Component/s: None
Labels:
None

Description

Optimization Proposal

In Kylin, the index storage is in Parquet format. When querying, if Spark reads Parquet data, it needs to read the Footer information first. In cases where there are many columns, reading the Footer can consume a considerable amount of time. Therefore, caching the Footer information can improve query performance.

Attachments

Issue Links

relates to

KYLIN-5846 upgrade spark version to 3.2.0-kylin-4.6.16.0

Closed

Activity

People

Assignee:: Pengfei Zhan

Reporter:: Zhimin Wu

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 31/Mar/24 09:37

Updated:: 12/Apr/24 09:41

Resolved:: 12/Apr/24 09:41