Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-4053

[Python/Integration] HDFS Tests failing with I/O operation on closed file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.12.0
    • Integration, Python

    Description

      To reporoduce run `docker-compose run hdfs-integration`

      self = <pyarrow.tests.test_hdfs.TestLibHdfs testMethod=test_cat>
      
          def test_cat(self):
              path = pjoin(self.tmp_path, 'cat-test')
      
              data = b'foobarbaz'
              with self.hdfs.open(path, 'wb') as f:
                  f.write(data)
      
      >       contents = self.hdfs.cat(path)
      
      opt/conda/lib/python3.6/site-packages/pyarrow/tests/test_hdfs.py:99:
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      opt/conda/lib/python3.6/site-packages/pyarrow/filesystem.py:42: in cat
          return f.read()
      pyarrow/io.pxi:298: in pyarrow.lib.NativeFile.read
          ???
      pyarrow/io.pxi:180: in pyarrow.lib.NativeFile.size
          ???
      pyarrow/io.pxi:142: in pyarrow.lib.NativeFile.get_random_access_file
          ???
      pyarrow/io.pxi:159: in pyarrow.lib.NativeFile._assert_readable
          ???
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
      >   ???
      E   ValueError: I/O operation on closed file
      

      Attachments

        Activity

          People

            wesm Wes McKinney
            kszucs Krisztian Szucs
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: