[ARROW-5049] [Python] org/apache/hadoop/fs/FileSystem class not found when pyarrow FileSystem used in spark

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0, 0.12.1, 0.13.0
• Fix Version/s: 0.14.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21543
• Assignee: Tiger068
• Reporter: Tiger068
• Created: 28/Mar/19 02:36
• Updated: 11/Jan/23 07:37
• Resolved: 25/Apr/19 20:08

Description

when i init pyarrow filesystem to connect hdfs clusfter in spark,the libhdfs throws error:

org/apache/hadoop/fs/FileSystem class not found 

I print out the CLASSPATH, the classpath value is wildcard mode  

../share/hadoop/hdfs;spark/spark-2.0.2-bin-hadoop2.7/jars...

The value is set by spark,but libhdfs must load class from jar files.

 

Root cause is:

In hdfs.py we just check the string  ''hadoop"  in classpath,but not jar file

def _maybe_set_hadoop_classpath():
    if 'hadoop' in os.environ.get('CLASSPATH', ''):
        return

Issue Links

• links to: GitHub Pull Request #4081