[GOBBLIN-422] FileBasedSource needs fs snapshot update of previously failed workunits with latest snapshot - ASF JIRA

XML

Word

Printable

JSON

Docs Text:

Hide
If the FileBasedSource initially starts processing 'source_dir' and finds 3 files underneath:

source_dir/file_1.txt
source_dir/file_2.txt
source_dir/file_3.txt

and we set:

source.filebased.maxFilesPerRun=2

If file_1.txt is corrupt and fails to be processed repeatedly, the workunit created for file_1.txt has the 'source.filebased.fs.snapshot' property set to file_1.txt,file_2.txt in the first run. Because of this, currently file_3.txt is repeatedly processed as long as file_1.txt fails to be processed in every run because the logic decides that file_3.txt is being processed for the first time using file_1.txt workunit's 'source.filebased.fs.snapshot' property.

Show
If the FileBasedSource initially starts processing 'source_dir' and finds 3 files underneath: source_dir/file_1.txt source_dir/file_2.txt source_dir/file_3.txt and we set: source.filebased.maxFilesPerRun=2 If file_1.txt is corrupt and fails to be processed repeatedly, the workunit created for file_1.txt has the 'source.filebased.fs.snapshot' property set to file_1.txt,file_2.txt in the first run. Because of this, currently file_3.txt is repeatedly processed as long as file_1.txt fails to be processed in every run because the logic decides that file_3.txt is being processed for the first time using file_1.txt workunit's 'source.filebased.fs.snapshot' property.