[LIVY-429] Livy skips url fragment identifier in spark.yarn.dist.archives setting

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.3
• Fix Version/s: 0.5.0
• Component/s: Server
• Assignee: Artem Plotnikov
• Reporter: Artem Plotnikov
• Created: 26/Dec/17 10:00
• Updated: 04/Jan/18 02:25
• Resolved: 04/Jan/18 02:24

Description

When submitting Spark session via Livy API with the following parameters:

{
	"kind": "pyspark",
	"name": "my-app"
	"conf": {
		"spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./ENVS/custom-python/bin/python"
	},
	"archives": ["/path/to/custom-python.zip#ENVS"]
}

Spark session fails with:

java.io.IOException: Cannot run program "./ENVS/custom-python/bin/python": error=2, No such file or directory

Because Livy uses java.net.URI#getPath method under the hood, which just skips URI fragment identifier part and Spark is being submitted with spark.yarn.dist.archives=/path/to/custom-python.zip and does not even extracts the archive in this case.