[ARROW-4053] [Python/Integration] HDFS Tests failing with I/O operation on closed file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Component/s: Integration, Python
• Labels: hdfs
• External issue URL: https://github.com/apache/arrow/issues/20651
• Assignee: Wes McKinney
• Reporter: Krisztian Szucs
• Created: 17/Dec/18 12:46
• Updated: 11/Jan/23 07:31
• Resolved: 18/Dec/18 18:25

Description

To reporoduce run `docker-compose run hdfs-integration`

self = <pyarrow.tests.test_hdfs.TestLibHdfs testMethod=test_cat>

    def test_cat(self):
        path = pjoin(self.tmp_path, 'cat-test')

        data = b'foobarbaz'
        with self.hdfs.open(path, 'wb') as f:
            f.write(data)

>       contents = self.hdfs.cat(path)

opt/conda/lib/python3.6/site-packages/pyarrow/tests/test_hdfs.py:99:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
opt/conda/lib/python3.6/site-packages/pyarrow/filesystem.py:42: in cat
    return f.read()
pyarrow/io.pxi:298: in pyarrow.lib.NativeFile.read
    ???
pyarrow/io.pxi:180: in pyarrow.lib.NativeFile.size
    ???
pyarrow/io.pxi:142: in pyarrow.lib.NativeFile.get_random_access_file
    ???
pyarrow/io.pxi:159: in pyarrow.lib.NativeFile._assert_readable
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   ValueError: I/O operation on closed file