[HUDI-1406] Add new DFS path sector implementation for listing date based partitions

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Component/s: deltastreamer
• Labels: pull-request-available
• Assignee: Bhavani Sudha
• Reporter: Bhavani Sudha
• Created: 19/Nov/20 21:28
• Updated: 21/Jan/21 06:02
• Resolved: 21/Jan/21 06:02

Description

Deltastreamer DFS source lists files from table path and determine files changed recently based on modification time. For certain workloads where only the latest partitions are affected, we might benefit by listing source input only from recent partitions. This especially helps data  in S3 with multi partition fields and  listing is time consuming. 

 

To support this, I propose adding a DFS selector implementation based on date partitions.

Issue Links

• links to: GitHub Pull Request #2264