Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1655

Support custom date format and fix unsupported exception in DatePartitionPathSelector

    XMLWordPrintableJSON

Details

    Description

      Add a config to allow parsing custom date format in DatePartitionPathSelector. Currently it assumes date partition string in the format of yyyy-MM-dd.

       

      Also eligibleFiles.sort() throws this exception

      java.lang.UnsupportedOperationException at java.util.AbstractList.set(AbstractList.java:132) at java.util.AbstractList$ListItr.set(AbstractList.java:426) at java.util.List.sort(List.java:482) at org.apache.hudi.utilities.sources.helpers.DatePartitionPathSelector.getNextFilePathsAndMaxModificationTime(DatePartitionPathSelector.java:141) at org.apache.hudi.utilities.sources.ParquetDFSSource.fetchNextBatch(ParquetDFSSource.java:48) at org.apache.hudi.utilities.sources.RowSource.fetchNewData(RowSource.java:43) at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:75) at org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter.fetchNewDataInRowFormat(SourceFormatAdapter.java:94) at org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:338) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:255) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:587)

       

      org.apache.hudi.client.common.HoodieSparkEngineContext#flatMap returns a list that can't be sorted in-place.

      Attachments

        Issue Links

          Activity

            People

              xushiyan Raymond Xu
              xushiyan Raymond Xu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: