Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.4.0
-
None
-
None
-
None
-
Linux 2.6.32-279.el6.x86_64
Description
Inadvertently,we set two source ftp path:
{ { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, FILE, null },pending,[(container_20150608111420_41540_1213_1503_)],4237640867118938,DOWNLOADING}
ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null },pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,DOWNLOADING}
the first one is a wrong path,only one source was set this;but Follow the log,i saw Starting from the first path source download,All next jobs sources were downloaded from ftp://10.27.178.207 by default.
the log is :
2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler] localizer.ResourceLocalizationService (ResourceLocalizationService.java:addResource(544)) - Downloading public rsrc:{ ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null } 2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler] localizer.ResourceLocalizationService (ResourceLocalizationService.java:addResource(544)) - Downloading public rsrc:{ ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, FILE, null } 2015-06-09 11:14:37,883 INFO [Public Localizer] localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(672)) - Failed to download rsrc { { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, FILE, null },pending,[(container_20150608111420_41540_1213_1503_)],4237640867118938,DOWNLOADING} java.io.IOException: Login failed on server - 10.27.178.207, port - 21 at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.java:133) at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:390) at com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.java:172) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:279) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:52) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-06-09 11:14:37,885 INFO [Public Localizer] localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource ftp://10.27.178.207:21/home/cbt/1213/jxf.sql transitioned from DOWNLOADING to FAILED 2015-06-09 11:14:37,886 INFO [Public Localizer] localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(672)) - Failed to download rsrc { { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null },pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,DOWNLOADING} java.io.IOException: Login failed on server - 10.27.178.207, port - 21 at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.java:133) at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:390) at com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.java:172) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:279) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.java:52) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-06-09 11:14:37,886 INFO [AsyncDispatcher event handler] container.Container (ContainerImpl.java:handle(853)) - Container container_20150608111420_41540_1213_1503_ transitioned from LOCALIZING to LOCALIZATION_FAILED 2015-06-09 11:14:37,887 INFO [Public Localizer] localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource ftp://10.27.89.13:21/home/cbt/common/2/sql.jar transitioned from DOWNLOADING to FAILED 2015-06-09 11:14:37,887 INFO [AsyncDispatcher event handler] localizer.LocalResourcesTrackerImpl (LocalResourcesTrackerImpl.java:handle(133)) - Container container_20150608111420_41540_1213_1503_ sent RELEASE event on a resource request { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null } not present
I debug the code of yarn.I found the piont is org.apache.hadoop.fs.FileSystem#cache
the code source is here:
private FileSystem getInternal(URI uri, Configuration conf, Key key) throws IOException{ FileSystem fs; synchronized (this) { fs = map.get(key); } if (fs != null) { return fs; } fs = createFileSystem(uri, conf); synchronized (this) { // refetch the lock again FileSystem oldfs = map.get(key); if (oldfs != null) { // a file system is created while lock is releasing fs.close(); // close the new file system return oldfs; // return the old file system } // now insert the new file system into the map if (map.isEmpty() && !ShutdownHookManager.get().isShutdownInProgress()) { ShutdownHookManager.get().addShutdownHook(clientFinalizer, SHUTDOWN_HOOK_PRIORITY); } fs.key = key; map.put(key, fs); if (conf.getBoolean("fs.automatic.close", true)) { toAutoClose.add(key); } return fs; } } ============================= FTPFileSystem.java @Override public void initialize(URI uri, Configuration conf) throws IOException { // get super.initialize(uri, conf); // get host information from uri (overrides info in conf) String host = uri.getHost(); host = (host == null) ? conf.get("fs.ftp.host", null) : host; if (host == null) { throw new IOException("Invalid host specified"); } conf.set("fs.ftp.host", host); // get port information from uri, (overrides info in conf) int port = uri.getPort(); port = (port == -1) ? FTP.DEFAULT_PORT : port; conf.setInt("fs.ftp.host.port", port); // get user/password information from URI (overrides info in conf) String userAndPassword = uri.getUserInfo(); if (userAndPassword == null) { userAndPassword = (conf.get("fs.ftp.user." + host, null) + ":" + conf .get("fs.ftp.password." + host, null)); if (userAndPassword == null) { throw new IOException("Invalid user/passsword specified"); } } String[] userPasswdInfo = userAndPassword.split(":"); conf.set("fs.ftp.user." + host, userPasswdInfo[0]); if (userPasswdInfo.length > 1) { conf.set("fs.ftp.password." + host, userPasswdInfo[1]); } else { conf.set("fs.ftp.password." + host, null); } setConf(conf); this.uri = uri; }
firstly.we have a source in ftp://10.27.89.13:21 then cache will store this key ,and fs.ftp.host=10.27.89.13 ,fs.ftp.user.10.27.89.13=XX and
fs.ftp.password.10.27.89.13 will be set in conf. secondly,the source in ftp:/10.27.178.207 come, this is not exit in cache. cache will store this key ,and ,and fs.ftp.host=10.27.178.207 ,fs.ftp.user.10.27.178.207=XX and
fs.ftp.password.10.27.178.207 will be set in conf. the key point is that :
/** * Connect to the FTP server using configuration parameters * * * @return An FTPClient instance * @throws IOException */ private FTPClient connect() throws IOException { FTPClient client = null; Configuration conf = getConf(); String host = conf.get("fs.ftp.host"); int port = conf.getInt("fs.ftp.host.port", FTP.DEFAULT_PORT); String user = conf.get("fs.ftp.user." + host); String password = conf.get("fs.ftp.password." + host); client = new FTPClient(); client.connect(host, port); int reply = client.getReplyCode(); if (!FTPReply.isPositiveCompletion(reply)) { throw new IOException("Server - " + host + " refused connection on port - " + port); } else if (client.login(user, password)) { client.setFileTransferMode(FTP.BLOCK_TRANSFER_MODE); client.setFileType(FTP.BINARY_FILE_TYPE); client.setBufferSize(DEFAULT_BUFFER_SIZE); } else { throw new IOException("Login failed on server - " + host + ", port - " + port); } return client; }
FTPFileSystem use conf to get host .port .username and password.
after the first two steps. the fs.ftp.host in conf is set to 10.27.178.207.
thirdly,a source in ftp://10.27.89.13:21 come.cache find there is exit.
so ftpClient use cache to connect. But the fs.ftp.host in conf is 10.27.178.207!!!! it's Confusing！！！
