[YARN-8690] Currently path not consistent in LocalResourceRequest to yarn 2.7

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.1
• Assignee: Wangda Tan
• Reporter: jialei weng
• Created: 21/Aug/18 07:20
• Updated: 21/Aug/18 07:26

Description

With YARN-1953 change,  in yarn 2.9.1, we can not use path, like hdfs://hostname/path, to local resource allocation as it will be resolved to hdfs://hostname:0/path. We have to add the port 443 in path, like hdfs://hostname:443/path, to make it work. It isn't a consistent change. If we can make it consistent  without customer's change? Wangda Tan

Handle resource location path in 2.7

Handle resource location logic in 2.9

public static Path getPathFromYarnURL(URL url) throws URISyntaxException {     String scheme = url.getScheme() == null ? "" : url.getScheme();         String authority = "";     if (url.getHost() != null) Unknown macro: {      authority = url.getHost();    if (url.getUserInfo() != null) {         authority = url.getUserInfo() + "@" + authority;             }      if (url.getPort() > 0) {        authority += ":" + url.getPort();      }    }         return new Path(         (new URI(scheme, authority, url.getFile(), null, null)).normalize());   }

public Path toPath() throws URISyntaxException {     return new Path(new URI(getScheme(), getUserInfo(),       getHost(), getPort(), getFile(), null, null));   }

Issue Links

• Blocked: YARN-1942 Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc.