Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7673

[C++][Dataset] Revisit File discovery failure mode

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • C++

    Description

      Currently, the default `FileSystemFactoryOptions::exclude_invalid_files` will silently ignore unsupported files (either IO error, not of the valid format, corruption, missing compression codecs, etc...) when creating a `FileSystemSource`.

      We should change this behavior to propagate an error in the Inspect/Finish calls by default and allow the user to toggle `exclude_invalid_files`. The error should contain at least the file path and a decipherable error (if possible).

      Attachments

        Issue Links

          Activity

            People

              fsaintjacques Francois Saint-Jacques
              fsaintjacques Francois Saint-Jacques
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: