[ARROW-9906] [Python] Crash in test_parquet.py::test_parquet_writer_filesystem_s3_uri (closing NativeFile from S3FileSystem)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: C++, Python
• Labels: filesystem pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25939
• Assignee: Antoine Pitrou
• Reporter: Joris Van den Bossche
• Created: 03/Sep/20 13:43
• Updated: 11/Jan/23 08:09
• Resolved: 09/Sep/20 10:07

Description

See https://github.com/apache/arrow/pull/7991#discussion_r481247263 and the commented out test added in that PR.

It doesn't give any clarifying traceback or crash message, but it segfaults when closing the NativeFile returned from S3FileSystem.open_output_stream in ParquetWriter.close().

With gdb I get a bit more context:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fa1a39df8f2 in arrow::fs::(anonymous namespace)::ObjectOutputStream::UploadPart (this=0x5619a95ce820, data=0x7fa197641ec0, nbytes=15671, owned_buffer=...) at ../src/arrow/filesystem/s3fs.cc:806
806       client_->UploadPartAsync(req, handler);

Another S3 crash in the parquet tests: ARROW-9814 (although it doesn't seem fully related)

Issue Links

• links to: GitHub Pull Request #8141