[ARROW-2002] use pyarrow download file will raise queue.Full exceptions sometimes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: Python
• Labels: pull-request-available
• Environment: operating system: all ; platform: all
• External issue URL: https://github.com/apache/arrow/issues/17984
• Assignee: kmiku7
• Reporter: kmiku7
• Created: 17/Jan/18 05:07
• Updated: 11/Jan/23 07:18
• Resolved: 17/Jan/18 17:26

Description

When we download file from hdfs, if the speed writer thread write data is slower than read speed, download() will raise queue.Fulll exceptions, because write_queue is full.

I think when we download file, we can wait until write_queue has space to enqueue new item if writer_thread is alive. Like what upload() does.

>>> import pyarrow as pa
>>> cli = pa.hdfs.connect(user='USERNAME')
>>> cli.download('/REMOTE/HDFS/PATH', '/LOCAL/FILE/PATH')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/io-hdfs.pxi", line 428, in pyarrow.lib.HadoopFileSystem.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:66399)
  File "pyarrow/io-hdfs.pxi", line 429, in pyarrow.lib.HadoopFileSystem.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:66351)
  File "pyarrow/io.pxi", line 315, in pyarrow.lib.NativeFile.download (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:52249)
  File "/usr/lib/python3.4/queue.py", line 187, in put_nowait
    return self.put(item, block=False)
  File "/usr/lib/python3.4/queue.py", line 133, in put
    raise Full
queue.Full

Issue Links

• links to: GitHub Pull Request #1485