Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1406

Add new DFS path sector implementation for listing date based partitions

    XMLWordPrintableJSON

    Details

      Description

      Deltastreamer DFS source lists files from table path and determine files changed recently based on modification time. For certain workloads where only the latest partitions are affected, we might benefit by listing source input only from recent partitions. This especially helps data  in S3 with multi partition fields and  listing is time consuming. 

       

      To support this, I propose adding a DFS selector implementation based on date partitions.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bhavanisudha Bhavani Sudha
                Reporter:
                bhavanisudha Bhavani Sudha
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: