[ARROW-4208] [CI/Python] Have automatized tests for S3

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: Continuous Integration, Python
• Labels: filesystem pull-request-available s3
• External issue URL: https://github.com/apache/arrow/issues/20788
• Assignee: Rok Mihevc
• Reporter: Krisztian Szucs
• Created: 09/Jan/19 12:56
• Updated: 11/Jan/23 07:32
• Resolved: 01/Oct/19 14:10

Description

Currently We don't run S3 integration tests regularly.

Possible solutions:

• mock it within python/pytest
• simply run the s3 tests with an S3 credential provided
• create a hdfs-integration like docker-compose setup and run an S3 mock server (e.g.: https://github.com/adobe/S3Mock, https://github.com/jubos/fake-s3, https://github.com/gaul/s3proxy, https://github.com/jserver/mock-s3)

For more see discussion https://github.com/apache/arrow/pull/3286

Issue Links

• Is contained by: ARROW-1119 [Python/C++] Implement NativeFile interfaces for Amazon S3
• relates to: ARROW-453 [C++] Add filesystem implementation for Amazon S3
• links to: GitHub Pull Request #5200