[IMPALA-13200] Auto refresh on S3 tables based on S3 notification

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 08/Jul/24 08:28
• Updated: 08/Jul/24 08:28

Description

S3 Event Notifications can be used to get updates on new files or file deletions:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

Snowflake uses it to auto refresh external tables:
https://docs.snowflake.com/en/user-guide/tables-external-s3 

Other object storages like Google Cloud Storage and Azure Blob Storage also have the similar notification mechanism:
https://docs.snowflake.com/en/user-guide/tables-external-gcs
https://docs.snowflake.com/en/user-guide/tables-external-azure

CC [e-mail] hemanth619 VenuReddy ngangam