Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2306

[Python] HDFS test failures

    XMLWordPrintableJSON

Details

    Description

      These weren't caught because we aren't running the HDFS tests in Travis CI

      pyarrow/tests/test_hdfs.py::TestLibHdfs::test_write_to_dataset_no_partitions FAILED
      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      
      self = <pyarrow.tests.test_hdfs.TestLibHdfs testMethod=test_write_to_dataset_no_partitions>
      
          @test_parquet.parquet
          def test_write_to_dataset_no_partitions(self):
              tmpdir = pjoin(self.tmp_path, 'write-no_partitions-' + guid())
              self.hdfs.mkdir(tmpdir)
              test_parquet._test_write_to_dataset_no_partitions(
      >           tmpdir, filesystem=self.hdfs)
      
      pyarrow/tests/test_hdfs.py:367: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      pyarrow/tests/test_parquet.py:1475: in _test_write_to_dataset_no_partitions
          filesystem=filesystem)
      pyarrow/parquet.py:1059: in write_to_dataset
          _mkdir_if_not_exists(fs, root_path)
      pyarrow/parquet.py:1006: in _mkdir_if_not_exists
          if fs._isfilestore() and not fs.exists(path):
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <pyarrow.hdfs.HadoopFileSystem object at 0x7f09e87a4c48>
      
          def _isfilestore(self):
              """
                  Returns True if this FileSystem is a unix-style file store with
                  directories.
                  """
      >       raise NotImplementedError
      E       NotImplementedError
      
      pyarrow/filesystem.py:143: NotImplementedError
      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      > /home/wesm/code/arrow/python/pyarrow/filesystem.py(143)_isfilestore()
      -> raise NotImplementedError
      (Pdb) c
      
      pyarrow/tests/test_hdfs.py::TestLibHdfs::test_write_to_dataset_with_partitions FAILED
      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      
      self = <pyarrow.tests.test_hdfs.TestLibHdfs testMethod=test_write_to_dataset_with_partitions>
      
          @test_parquet.parquet
          def test_write_to_dataset_with_partitions(self):
              tmpdir = pjoin(self.tmp_path, 'write-partitions-' + guid())
              self.hdfs.mkdir(tmpdir)
              test_parquet._test_write_to_dataset_with_partitions(
      >           tmpdir, filesystem=self.hdfs)
      
      pyarrow/tests/test_hdfs.py:360: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      pyarrow/tests/test_parquet.py:1433: in _test_write_to_dataset_with_partitions
          filesystem=filesystem)
      pyarrow/parquet.py:1059: in write_to_dataset
          _mkdir_if_not_exists(fs, root_path)
      pyarrow/parquet.py:1006: in _mkdir_if_not_exists
          if fs._isfilestore() and not fs.exists(path):
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <pyarrow.hdfs.HadoopFileSystem object at 0x7f09e87a4c48>
      
          def _isfilestore(self):
              """
                  Returns True if this FileSystem is a unix-style file store with
                  directories.
                  """
      >       raise NotImplementedError
      E       NotImplementedError
      
      pyarrow/filesystem.py:143: NotImplementedError
      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      > /home/wesm/code/arrow/python/pyarrow/filesystem.py(143)_isfilestore()
      -> raise NotImplementedError
      (Pdb) c
      

      Attachments

        Issue Links

          Activity

            People

              wesm Wes McKinney
              wesm Wes McKinney
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: