Mayank Bansal Are you still looking into this issue? or else I would like to take over.

Attaching Patch

Thanks,
Mayank

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12574430/YARN-109-trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

-1 one of tests included doesn't have a timeout.

-1 javac. The applied patch generated 1362 javac compiler warnings (more than the trunk's current 1361 warnings).

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/544//testReport/
Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/544//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/544//console

This message is automatically generated.

Fixing Warning

Mayank Bansal can we move the file deletion logic to the unpack method? I think that would be the right place to delete the file as it is already happening for resource.getType() == FILE. (during renameTo call). The current patch won't handle the situation where
1) user has specified the file type as ARCHIVE but the extension is neither of these (.jar,.tar,.tar.gz,.tgz) then it will add an incorrect error log message about file not deleted.
2) If the user has specified the file type as PATTERN then this will not remove the file.

Let me know what are your thoughts.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12574444/YARN-109-trunk-1.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

-1 one of tests included doesn't have a timeout.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/545//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/545//console

This message is automatically generated.

Omkar Vinit Joshi
Good points.
Adding another patch.

Thanks,
Mayank

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12574666/YARN-109-trunk-2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

-1 one of tests included doesn't have a timeout.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/551//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/551//console

This message is automatically generated.

Mayank Bansal I looked at the patch. Even though it is not modifying call method logically but still I see changes there.

Can we simply keep a flag to track all the places in unpack method where renameTo was called (To indicate not to delete file as it is already deleted). and towards the end before returning we can delete file based on flag value. For file deletion we can even use File.delete(). I guess this will make it cleaner and will keep the tmp file handling logic intact as we need to log an error message in case we are unable to delete the .tmp file.

The code is still not handling situation where resource.getType() == ARCHIVE and it is .jar or if it is PATTERN and .zip file.

Also I see that entire test code is only testing if the file is a tar file and ARCHIVE type but not for other extensions / TYPE. I think we need to test all the combinations ( .zip , .tar, .jar ) and it is specified as FILE, PATTERN or ARCHIVE.

Also for the test I think you will have to add a timeout parameter. @Test(timeout=<value>) to avoid -1.

Please let me know your thoughts on this.

Omkar Vinit Joshi
There are no changes in call its just I formatted that with hadoop formatter.

I modified the unpack for all the cases.

I added couple of more test methods to test some more cases , Looks to me enough for this change. Not necessary to test all combinations.

I add the timeouts.

Thanks,
Mayank

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12574743/YARN-109-trunk-3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

-1 one of tests included doesn't have a timeout.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/555//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/555//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/555//console

This message is automatically generated.

The findbugs is complaining that you are ignoring the return value of the delete call. It should not be a problem so either use the return value to log a warning when it fails or update the findbugs filter to filter out the error.

The -1 for the test timeouts is caused by a bug in the script used to detect these, so you can either ignore it, or add a timeout to any @Test that appears in the patch file, including the ones you didn't add .

In the test, please uncomment the lines to cleanup after the test. Are they causing a problem for the test to pass? or was it just for debugging?

Also I personally would prefer to have a few small jar/tar/zip files checked into the repository instead of generating them on they fly for the test. It will speed up the test and have less dependencies on the system being set up with the exact commands, i.e. bash for windows support. Although if you don't feel like changing it I am fine with that too, most of those commands are used by FSDownload already so it is not that critical.

Mayank Bansal Thanks for updating it. Now it looks good.

Robert Joseph Evans

Thanks for the comments and review. Updating the new patch.

For adding the jar/tar/zip files in to repo I am not sure at this moment as i see in other cases it is also generating it on the fly.
However I think your point is valid.
I will recommend to do that in separate JIRA for everything to maintain consistency.

Thoughts?

Thanks,
Mayank

That is fine with me. My concern was mostly with Windows support. tar, zip, jar, etc. should be there, but bash may not be. So if you want to file a new JIRA that is fine, if not you can just wait for windows support to be merged in and see if it breaks.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12574885/YARN-109-trunk-4.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 tests included appear to have a timeout.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.yarn.util.TestFSDownload

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/559//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/559//console

This message is automatically generated.

Robert Joseph Evans
Sure I will file a new JIRA for that.

Thanks,
Mayank

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12574891/YARN-109-trunk-5.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 tests included appear to have a timeout.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/563//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/563//console

This message is automatically generated.

Robert Joseph Evans
Thanks for the review.
Do you mind committing it as well?

Thanks,
Mayank

The changes look good to me. The test seems a bit brittle, looking to be sure that a .tmp is not there, but I don't know any other way to do it. +1

I'll check it in.

Thanks Mayank,

I put this into trunk, branch-2, and branch-0.23. Keep up the good work.

Integrated in Hadoop-trunk-Commit #3521 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3521/)
YARN-109. .tmp file is not deleted for localized archives (Mayank Bansal via bobby) (Revision 1460723)

Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1460723
Files :

• /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java
• /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java

Integrated in Hadoop-Yarn-trunk #167 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/167/)
YARN-109. .tmp file is not deleted for localized archives (Mayank Bansal via bobby) (Revision 1460723)

Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1460723
Files :

• /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java
• /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java

Integrated in Hadoop-Hdfs-0.23-Build #565 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/565/)
svn merge -c 1460723 FIXES: YARN-109. .tmp file is not deleted for localized archives (Mayank Bansal via bobby) (Revision 1460734)

Result = UNSTABLE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1460734
Files :

• /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java
• /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java

Integrated in Hadoop-Hdfs-trunk #1356 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1356/)
YARN-109. .tmp file is not deleted for localized archives (Mayank Bansal via bobby) (Revision 1460723)

Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1460723
Files :

• /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java
• /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java

Integrated in Hadoop-Mapreduce-trunk #1384 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1384/)
YARN-109. .tmp file is not deleted for localized archives (Mayank Bansal via bobby) (Revision 1460723)

Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1460723
Files :

• /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java
• /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java