[MESOS-9348] URL-encoded HDFS artifacts can't be fetched through the cache.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: James Peach
• Created: 22/Oct/18 23:46
• Updated: 23/Oct/18 00:13

Description

The hdfs dfs command always does a URI decode on the target output file. This means that the output file gets stored in the fetcher cache under the wrong filename and we can never retrieve it.

Here's an example of how the command behaves:

[/tmp]# hdfs dfs -copyToLocal hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar $(pwd)/%5B-jpeach-%5D.jar

[/tmp]# ls -l *jpeach*
-rw-r--r-- 1 root root 7285799 Oct 22 23:29 [-jpeach-].jar

Here's how this plays out in the fetcher:

W1022 23:22:13.649587 3186459 fetcher.cpp:395] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar
cp: cannot stat `/srv/mesos/fetch/jarvis/c67-connector-_ASE%5D.jar': No such file or directory
E1022 23:22:13.652987 3186459 fetcher.cpp:613] EXIT with status 1: Failed to fetch 'hdfs:///artifacts/8c/99/4b/8c994b489674589a58805e2e695e98674b9dd793411579f0fbaea3459f94f86e/connector/%5BRELEASE%5D/connector-%5BRELEASE%5D.jar': cp failed with status: 256
...
# ls -latr /srv/mesos/fetch
...
-rw-r--r-- 1 jarvis jarvis   7285799 Oct 22 23:22 c67-connector-_ASE].jar

The fetcher has downloaded the artifact into the cache, but can't copy it into the sandbox because it was downloaded to the wrong filename.