Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-10012

[C++] Sporadic failures in CopyFiles test

    XMLWordPrintableJSON

    Details

      Description

      Here is an example failure on AppVeyor (Windows):
      https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/35210915/job/yhdxkfu7hta3kugp#L1913

      [ RUN      ] TestSubTreeFileSystem.CopyFiles
      C:/projects/arrow/cpp/src/arrow/filesystem/filesystem_test.cc(608): error: Failed
      'CopyFiles({{subfs_, "ab"}, {subfs_, "cd"}, {subfs_, "ef"}}, {{dest_fs, "AB/ab"}, {dest_fs, "CD/CD/cd"}, {dest_fs, "EF/EF/EF/ef"}})' failed with IOError: Path does not exist 'sub/copy/AB/ab'
      [  FAILED  ] TestSubTreeFileSystem.CopyFiles (9 ms)
      

      Similar failure on Linux, showing the line number:
      https://ci.ursalabs.org/#/builders/101/builds/4412/steps/8/logs/stdio

      [ RUN      ] TestSubTreeFileSystem.CopyFiles
      ../src/arrow/filesystem/filesystem_test.cc:608: Failure
      Failed
      'CopyFiles({{subfs_, "ab"}, {subfs_, "cd"}, {subfs_, "ef"}}, {{dest_fs, "AB/ab"}, {dest_fs, "CD/CD/cd"}, {dest_fs, "EF/EF/EF/ef"}})' failed with IOError: Path does not exist 'sub/copy/CD/CD/cd'
      In ../src/arrow/filesystem/filesystem.cc, line 467, code: (_error_or_value13).status()
      [  FAILED  ] TestSubTreeFileSystem.CopyFiles (17 ms)
      

      The erroring line number (467) corresponds to the OpenOutputStream call below:

      
              auto dest_dir = internal::GetAbstractPathParent(destinations[i].path).first;
              if (!dest_dir.empty()) {
                RETURN_NOT_OK(destinations[i].filesystem->CreateDir(dest_dir));
              }
      
      -->     ARROW_ASSIGN_OR_RAISE(
                  auto destination,
                  destinations[i].filesystem->OpenOutputStream(destinations[i].path));
      

      It's not obvious why the error occurs, since the directory is created just above (race condition in the CreateDir implementation?).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                apitrou Antoine Pitrou
                Reporter:
                apitrou Antoine Pitrou
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m