[MESOS-9008] Fetcher fails to extract some archives containing hardlinks

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.7.0
• Component/s: fetcher
• Labels: mesosphere
• Target Version/s: 1.7.0
• Sprint: Mesosphere Sprint 2018-22, Mesosphere Sprint 2018-23
• Story Points: 1
• Assignee: Joseph Wu
• Reporter: Benjamin Bannier
• Shepherd: Andrew Schwartzmeyer
• Created: 19/Jun/18 16:49
• Updated: 21/Jun/18 21:14
• Resolved: 21/Jun/18 21:14

Description

We recently switched the infrastructure to e.g., extract archives to libarchive which seems to have narrower support for e.g., hardlinks in archives, see e.g., https://github.com/libarchive/libarchive/wiki/Hardlinks upstream (likely outdated).

In a particular case, we tried to extract https://artifacts.elastic.co/downloads/kibana/kibana-5.6.5-linux-x86_64.tar.gz which the fetcher successfully extracted in e.g., 1.6.0, but which now leads to failures like

W0619 08:53:38.000000  4610 fetcher.cpp:913] Begin fetcher log (stderr in sandbox) for container d39bc16c-e6c6-440f-a82c-ad26332a1b36 from running command: /opt/mesosphere/packages/mesos--95f27ab971fb1d03bc1277f6a16e63a9815e1a61/libexec/mesos/mesos-fetcher
I0619 08:53:34.620458  7571 fetcher.cpp:560] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/nobody","items":[{"action":"DOWNLOAD_AND_CACHE","cache_filename":"c29-kibana-5.6__64.tar.gz","uri":{"cache":true,"executable":false,"extract":true,"value":"https:\/\/artifacts.elastic.co\/downloads\/kibana\/kibana-5.6.5-linux-x86_64.tar.gz"}},{"action":"DOWNLOAD_AND_CACHE","cache_filename":"c30-x-pack-5.6.5.zip","uri":{"cache":true,"executable":false,"extract":false,"value":"https:\/\/artifacts.elastic.co\/downloads\/packs\/x-pack\/x-pack-5.6.5.zip"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/7f2e186c-c748-493d-82e4-644b5b23c9bb-S6\/frameworks\/7f2e186c-c748-493d-82e4-644b5b23c9bb-0001\/executors\/kibana.3f6c7799-739e-11e8-a678-168288e5aa33\/runs\/d39bc16c-e6c6-440f-a82c-ad26332a1b36","stall_timeout":{"nanoseconds":60000000000},"user":"nobody"}
I0619 08:53:34.623996  7571 fetcher.cpp:457] Fetching URI 'https://artifacts.elastic.co/downloads/kibana/kibana-5.6.5-linux-x86_64.tar.gz'
I0619 08:53:34.624017  7571 fetcher.cpp:431] Downloading into cache
I0619 08:53:34.624027  7571 fetcher.cpp:225] Fetching URI 'https://artifacts.elastic.co/downloads/kibana/kibana-5.6.5-linux-x86_64.tar.gz'
I0619 08:53:34.624043  7571 fetcher.cpp:175] Downloading resource from 'https://artifacts.elastic.co/downloads/kibana/kibana-5.6.5-linux-x86_64.tar.gz' to '/tmp/mesos/fetch/nobody/c29-kibana-5.6__64.tar.gz'
I0619 08:53:38.634416  7571 fetcher.cpp:350] Fetching from cache
E0619 08:53:38.686941  7571 fetcher.cpp:613] EXIT with status 1: Failed to fetch 'https://artifacts.elastic.co/downloads/kibana/kibana-5.6.5-linux-x86_64.tar.gz': Failed to extract archive '/tmp/mesos/fetch/nobody/c29-kibana-5.6__64.tar.gz' to '/var/lib/mesos/slave/slaves/7f2e186c-c748-493d-82e4-644b5b23c9bb-S6/frameworks/7f2e186c-c748-493d-82e4-644b5b23c9bb-0001/executors/kibana.3f6c7799-739e-11e8-a678-168288e5aa33/runs/d39bc16c-e6c6-440f-a82c-ad26332a1b36': Failed to write archive header: Hard-link target 'kibana-5.6.5-linux-x86_64/node_modules/svgo/node_modules/js-yaml/bin/js-yaml.js' does not exist.

End fetcher log for container d39bc16c-e6c6-440f-a82c-ad26332a1b36
E0619 08:53:38.000000  4610 fetcher.cpp:571] Failed to run mesos-fetcher: Failed to fetch all URIs for container 'd39bc16c-e6c6-440f-a82c-ad26332a1b36': exited with status 1
E0619 08:53:38.000000  4608 slave.cpp:6180] Container 'd39bc16c-e6c6-440f-a82c-ad26332a1b36' for executor 'kibana.3f6c7799-739e-11e8-a678-168288e5aa33' of framework 7f2e186c-c748-493d-82e4-644b5b23c9bb-0001 failed to start: Failed to fetch all URIs for container 'd39bc16c-e6c6-440f-a82c-ad26332a1b36': exited with status 1