[ARROW-4413] [Python] pyarrow.hdfs.connect() failing

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.13.0
• Component/s: Python
• Labels: pull-request-available
• Environment: Python 2.7 ; Hadoop distribution: Amazon 2.7.3 ; Hive 2.1.1 ; Spark 2.1.1 ; Tez 0.8.4 ; Linux 4.4.35-33.55.amzn1.x86_64
• External issue URL: https://github.com/apache/arrow/issues/20975
• Assignee: Antoine Pitrou
• Reporter: Bradley Grantham
• Created: 29/Jan/19 09:03
• Updated: 11/Jan/23 07:33
• Resolved: 19/Mar/19 18:09

Description

Trying to connect to hdfs using the below snippet. Using hadoop-libhdfs.
This error appears in v0.12.0. It doesn't appear in v0.11.1. (I used the same environment when testing that it still worked on v0.11.1)

 

In [1]: import pyarrow as pa

In [2]: fs = pa.hdfs.connect()

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-e0007ad7fa95> in <module>()
----> 1 fs = pa.hdfs.connect()

/usr/local/lib64/python2.7/site-packages/pyarrow/hdfs.pyc in connect(host, port, user, kerb_ticket, driver, extra_conf)
    205     fs = HadoopFileSystem(host=host, port=port, user=user,
    206                           kerb_ticket=kerb_ticket, driver=driver,
--> 207                           extra_conf=extra_conf)
    208     return fs

/usr/local/lib64/python2.7/site-packages/pyarrow/hdfs.pyc in __init__(self, host, port, user, kerb_ticket, driver, extra_conf)
     36             _maybe_set_hadoop_classpath()
     37 
---> 38         self._connect(host, port, user, kerb_ticket, driver, extra_conf)
     39 
     40     def __reduce__(self):

/usr/local/lib64/python2.7/site-packages/pyarrow/io-hdfs.pxi in pyarrow.lib.HadoopFileSystem._connect()
     72         if host is not None:
     73             conf.host = tobytes(host)
---> 74         self.host = host
     75 
     76         conf.port = port

TypeError: Expected unicode, got str

Issue Links

• is duplicated by: ARROW-4943 pyarrow.lib.HadoopFileSystem._connect failed due to TypeError
• links to: GitHub Pull Request #3975