[SPARK-2881] Snappy is now default codec - could lead to conflicts since uses /tmp

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.0, 1.2.0
• Component/s: Spark Core
• Assignee: Patrick Wendell
• Reporter: Thomas Graves
• Created: 06/Aug/14 17:08
• Updated: 01/Sep/14 05:49
• Resolved: 18/Aug/14 00:07

Description

I was using spark master branch and I ran into an issue with Snappy since its now the default codec for shuffle.

The issue was that someone else had run with snappy and it created /tmp/snappy-*.so but it had restrictive permissions so I was not able to use it or remove it. This caused my spark job to not start.

I was running in yarn client mode at the time. Yarn cluster mode shouldn't have this issue since we change the java.io.tmpdir.
I assume this would also affect standalone mode.

I'm not sure if this is a true blocker but wanted to file it as one at first and let us decide.

Issue Links

• relates to: HIVE-7916 Snappy-java error when running hive query on spark [Spark Branch]
• links to: [Github] Pull Request #1991 (pwendell)
• links to: [Github] Pull Request #1994 (pwendell)
• links to: [Github] Pull Request #1995 (pwendell)
• links to: [Github] Pull Request #1999 (pwendell)