Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.1.0
-
None
Description
Today, we compare the whole path when deciding if a file is new in the FileSource for structured streaming. However, this cause cause false negatives in the case where the path has changed in a cosmetic way (i.e. changing s3n to s3a). We should add an option fileNameOnly that causes the new file check to be based only on the filename (but still store the whole path in the log).
