[MESOS-3602] hdfs du fails due to prepended / on path

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.24.2, 0.25.1, 0.26.0
• Component/s: fetcher
• Assignee: James Peach
• Reporter: alexius ludeman
• Created: 07/Oct/15 21:22
• Updated: 27/Feb/16 01:21
• Resolved: 04/Nov/15 12:11

Description

hdfs.hpp du() fails to run. It appears to prepend "/" but the path passed in is a uri of something like "hdfs:///a/path/to/artifact.tar.gz".

W1007 13:46:25.791894 373116928 fetcher.cpp:436] Reverting to fetching directly into the sandbox for 'hdfs:///a/path/to/3.3.2-SNAPSHOT/executor-3.3.2-SNAPSHOT-artifact-with-dependencies-archive.tar.gz', due to failure to fetch through the cache, with error: Could not determine size of cache file for 'lexinator@hdfs:///a/path/to/3.3.2-SNAPSHOT/executor-3.3.2-SNAPSHOT-artifact-with-dependencies-archive.tar.gz' with error: Hadoop client could not determine size: HDFS du returned an unexpected number of results: '2015-10-07 13:46:21,958 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-du: java.net.URISyntaxException: Expected scheme-specific part at index 5: hdfs:
Usage: hadoop fs [generic options] -du [-s] [-h] <path> ...

The command it's running is:

/usr/bin/env bash /.../hadoop fs -du -h /hdfs:///a/path/to/3.3.2-SNAPSHOT/executor-3.3.2-SNAPSHOT-artifact-with-dependencies-archive.tar.gz

Issue Links

• is depended upon by: MESOS-3605 hdfs.du() fails on os x due to lack of native-hadoop library.
• is required by: MESOS-3605 hdfs.du() fails on os x due to lack of native-hadoop library.