[ARROW-2306] [Python] HDFS test failures

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/15682
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 13/Mar/18 20:51
• Updated: 11/Jan/23 07:20
• Resolved: 14/Mar/18 10:20

Description

These weren't caught because we aren't running the HDFS tests in Travis CI

pyarrow/tests/test_hdfs.py::TestLibHdfs::test_write_to_dataset_no_partitions FAILED
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

self = <pyarrow.tests.test_hdfs.TestLibHdfs testMethod=test_write_to_dataset_no_partitions>

    @test_parquet.parquet
    def test_write_to_dataset_no_partitions(self):
        tmpdir = pjoin(self.tmp_path, 'write-no_partitions-' + guid())
        self.hdfs.mkdir(tmpdir)
        test_parquet._test_write_to_dataset_no_partitions(
>           tmpdir, filesystem=self.hdfs)

pyarrow/tests/test_hdfs.py:367: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/tests/test_parquet.py:1475: in _test_write_to_dataset_no_partitions
    filesystem=filesystem)
pyarrow/parquet.py:1059: in write_to_dataset
    _mkdir_if_not_exists(fs, root_path)
pyarrow/parquet.py:1006: in _mkdir_if_not_exists
    if fs._isfilestore() and not fs.exists(path):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyarrow.hdfs.HadoopFileSystem object at 0x7f09e87a4c48>

    def _isfilestore(self):
        """
            Returns True if this FileSystem is a unix-style file store with
            directories.
            """
>       raise NotImplementedError
E       NotImplementedError

pyarrow/filesystem.py:143: NotImplementedError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/wesm/code/arrow/python/pyarrow/filesystem.py(143)_isfilestore()
-> raise NotImplementedError
(Pdb) c

pyarrow/tests/test_hdfs.py::TestLibHdfs::test_write_to_dataset_with_partitions FAILED
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

self = <pyarrow.tests.test_hdfs.TestLibHdfs testMethod=test_write_to_dataset_with_partitions>

    @test_parquet.parquet
    def test_write_to_dataset_with_partitions(self):
        tmpdir = pjoin(self.tmp_path, 'write-partitions-' + guid())
        self.hdfs.mkdir(tmpdir)
        test_parquet._test_write_to_dataset_with_partitions(
>           tmpdir, filesystem=self.hdfs)

pyarrow/tests/test_hdfs.py:360: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/tests/test_parquet.py:1433: in _test_write_to_dataset_with_partitions
    filesystem=filesystem)
pyarrow/parquet.py:1059: in write_to_dataset
    _mkdir_if_not_exists(fs, root_path)
pyarrow/parquet.py:1006: in _mkdir_if_not_exists
    if fs._isfilestore() and not fs.exists(path):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyarrow.hdfs.HadoopFileSystem object at 0x7f09e87a4c48>

    def _isfilestore(self):
        """
            Returns True if this FileSystem is a unix-style file store with
            directories.
            """
>       raise NotImplementedError
E       NotImplementedError

pyarrow/filesystem.py:143: NotImplementedError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/wesm/code/arrow/python/pyarrow/filesystem.py(143)_isfilestore()
-> raise NotImplementedError
(Pdb) c

Issue Links

• links to: GitHub Pull Request #1742