[MESOS-3605] hdfs.du() fails on os x due to lack of native-hadoop library.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.24.2, 0.25.1, 0.26.1, 0.27.0
• Component/s: fetcher

Environment

os x

• Assignee: James Peach
• Reporter: alexius ludeman
• Created: 08/Oct/15 05:00
• Updated: 26/Apr/17 16:45
• Resolved: 16/Nov/15 09:10

Description

hdfs.du() fails on os x due to lack of native-hadoop library.

This requires a fix from https://issues.apache.org/jira/browse/MESOS-3602 before it's reproducible.

per https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html OS X does not have native library support, and I could not readily find a way to disable the message. This causes hdfs.du() to fail when parsing the output with the extra warning message about the unavailable native-hadoop library.

W1007 21:47:38.362117 250429440 fetcher.cpp:451] Reverting to fetching directly into the sandbox for 'hdfs:///a/path/artifacts/executor/3.3.2-SNAPSHOT/executor-3.3.2-SNAPSHOT-artifact-with-dependencies-archive.tar.gz', due to failure to fetch through the cache, with error: Could not determine size of cache file for 'lexinator@hdfs:///a/path/artifacts/executor/3.3.2-SNAPSHOT/executor-3.3.2-SNAPSHOT-artifact-with-dependencies-archive.tar.gz' with error: Hadoop client could not determine size: HDFS du returned an unexpected number of results: '2015-10-07 21:47:37,474 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
10.4 M  /a/path/artifacts/executor/3.3.2-SNAPSHOT/executor-3.3.2-SNAPSHOT-artifact-with-dependencies-archive.tar.gz

Issue Links

• depends upon: MESOS-3602 hdfs du fails due to prepended / on path
• relates to: MESOS-4283 Accept 3-field version of HDFS du output
• requires: MESOS-3602 hdfs du fails due to prepended / on path