I wonder if MAPREDUCE-2415 broke this.

It would get cleared even if a TT is sent a reinitialization signal from the JT too. We can perhaps selectively ignore "uesrlogs/" when deleting the local dirs and leave it to the userlog-manager to clean them up per the retention policy?

No, MAPREDUCE-2415 should not have broken this:

   public void cleanupStorage() throws IOException {
-    this.fConf.deleteLocalFiles();
+    this.fConf.deleteLocalFiles(SUBDIR);
+    this.fConf.deleteLocalFiles(TT_PRIVATE_DIR);
+    this.fConf.deleteLocalFiles(TT_LOG_TMP_DIR);
   }

As we can see there, it now deletes specific sub-dirs instead of all of them, when cleaning the local dirs.

What is the use case?

Arun - Isn't this a bug? Restarts of a service should not break the log retention guarantee?

I could not reproduce this on 1.0.3. I restarted the TT, re-initialized it (via JT restart), but the MR local directory userlogs/ still exist and continue to be symlinked inside the logs dir (and nowhere in code can I find a full clean delete of TaskLog.USERLOGS_DIR_NAME presently). Can you post some reproduce steps for this?

Sure, I will add the steps to reproduce.

This issue occurs only in those distributions where MAPREDUCE-2415 is applied as well as MRAsyncDiskService is used to cleanup the volumes during TT startup.
This is not applicable to 1.0 or 1.1 since MRAsyncDiskService is not present in those.
I don't think , it is applicable to trunk.

Thanks very much Benoy!

Reopen reason: This should still affect 0.22 due to MAPREDUCE-1213 being in it. Lets try to fix it for 0.22.x.

This will not impact 0.22. MAPREDUCE -2415 was not ported to 0.22 . So userlogs directory will not be under the scratch directories.

Ah yes, my bad. Would 0.22 benefit from MAPREDUCE-2415 though? If its not needed, we can probably close this out again.

Good point related to porting MAPREDUCE-2415 to 0.22.
Another related question will be porting MAPREDUCE-1213 to 1.1