Description
There are two bugs here.
1. The compareFs() method in ClientBase considers the 'host' part of the URI to be an actual host. In the case of HA and Federation, that's a namespace name, which doesn't resolve to anything. So in those cases, compareFs() always says the file systems are different.
2. In prepareLocalResources(), when adding a file to the distributed cache, that is done with the common FileSystem object instantiated at the start of the method. In the case of Federation that doesn't work: the qualified URL's scheme may differ from the non-qualified one, so the FileSystem instance will not work.
Fixes are pretty trivial.
Attachments
Issue Links
- is duplicated by
-
SPARK-4712 uploading jar when set spark.yarn.jar
-
- Closed
-
- links to
