Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-8283

[Python][Dataset] Non-existent files are silently dropped in pa.dataset.FileSystemDataset

    XMLWordPrintableJSON

Details

    Description

      When passing a list of files to the constructor of pyarrow.dataset.FileSystemDataset, all files that don't exist are silently dropped immediately (i.e. no fragments are created for them).

      Instead, I would expect that fragments will be created for them but an error is thrown when one tries to read the fragment with the non-existent file.

      Attachments

        Issue Links

          Activity

            People

              jorisvandenbossche Joris Van den Bossche
              uwe Uwe Korn
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m