Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-4413

[Python] pyarrow.hdfs.connect() failing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: Python
    • Environment:
      Python 2.7
      Hadoop distribution: Amazon 2.7.3
      Hive 2.1.1
      Spark 2.1.1
      Tez 0.8.4
      Linux 4.4.35-33.55.amzn1.x86_64

      Description

      Trying to connect to hdfs using the below snippet. Using hadoop-libhdfs.
      This error appears in v0.12.0. It doesn't appear in v0.11.1. (I used the same environment when testing that it still worked on v0.11.1)

       

      In [1]: import pyarrow as pa
      
      In [2]: fs = pa.hdfs.connect()
      
      ---------------------------------------------------------------------------
      TypeError                                 Traceback (most recent call last)
      <ipython-input-2-e0007ad7fa95> in <module>()
      ----> 1 fs = pa.hdfs.connect()
      
      /usr/local/lib64/python2.7/site-packages/pyarrow/hdfs.pyc in connect(host, port, user, kerb_ticket, driver, extra_conf)
          205     fs = HadoopFileSystem(host=host, port=port, user=user,
          206                           kerb_ticket=kerb_ticket, driver=driver,
      --> 207                           extra_conf=extra_conf)
          208     return fs
      
      /usr/local/lib64/python2.7/site-packages/pyarrow/hdfs.pyc in __init__(self, host, port, user, kerb_ticket, driver, extra_conf)
           36             _maybe_set_hadoop_classpath()
           37 
      ---> 38         self._connect(host, port, user, kerb_ticket, driver, extra_conf)
           39 
           40     def __reduce__(self):
      
      /usr/local/lib64/python2.7/site-packages/pyarrow/io-hdfs.pxi in pyarrow.lib.HadoopFileSystem._connect()
           72         if host is not None:
           73             conf.host = tobytes(host)
      ---> 74         self.host = host
           75 
           76         conf.port = port
      
      TypeError: Expected unicode, got str
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                apitrou Antoine Pitrou
                Reporter:
                bradleygrantham Bradley Grantham
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h