Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2002

use pyarrow download file will raise queue.Full exceptions sometimes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.8.0
    • 0.9.0
    • Python
    • operating system: all
      platform: all

    Description

      When we download file from hdfs, if the speed writer thread write data is slower than read speed, download() will raise queue.Fulll exceptions, because write_queue is full.

      I think when we download file, we can wait until write_queue has space to enqueue new item if writer_thread is alive. Like what upload() does.

      >>> import pyarrow as pa
      >>> cli = pa.hdfs.connect(user='USERNAME')
      >>> cli.download('/REMOTE/HDFS/PATH', '/LOCAL/FILE/PATH')
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "pyarrow/io-hdfs.pxi", line 428, in pyarrow.lib.HadoopFileSystem.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:66399)
        File "pyarrow/io-hdfs.pxi", line 429, in pyarrow.lib.HadoopFileSystem.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:66351)
        File "pyarrow/io.pxi", line 315, in pyarrow.lib.NativeFile.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:52249)
        File "/usr/lib/python3.4/queue.py", line 187, in put_nowait
          return self.put(item, block=False)
        File "/usr/lib/python3.4/queue.py", line 133, in put
          raise Full
      queue.Full
      

      Attachments

        Issue Links

          Activity

            People

              kmiku7 kmiku7
              kmiku7 kmiku7
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: