Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2002

use pyarrow download file will raise queue.Full exceptions sometimes

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.9.0
    • Component/s: Python
    • Environment:
      operating system: all
      platform: all

      Description

      When we download file from hdfs, if the speed writer thread write data is slower than read speed, download() will raise queue.Fulll exceptions, because write_queue is full.

      I think when we download file, we can wait until write_queue has space to enqueue new item if writer_thread is alive. Like what upload() does.

      >>> import pyarrow as pa
      >>> cli = pa.hdfs.connect(user='USERNAME')
      >>> cli.download('/REMOTE/HDFS/PATH', '/LOCAL/FILE/PATH')
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "pyarrow/io-hdfs.pxi", line 428, in pyarrow.lib.HadoopFileSystem.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:66399)
        File "pyarrow/io-hdfs.pxi", line 429, in pyarrow.lib.HadoopFileSystem.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:66351)
        File "pyarrow/io.pxi", line 315, in pyarrow.lib.NativeFile.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:52249)
        File "/usr/lib/python3.4/queue.py", line 187, in put_nowait
          return self.put(item, block=False)
        File "/usr/lib/python3.4/queue.py", line 133, in put
          raise Full
      queue.Full
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kmiku7 kmiku7
                Reporter:
                kmiku7 kmiku7
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: