[MAPREDUCE-6680] JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
• Component/s: jobhistoryserver
• Labels: Azure S3
• Target Version/s: 2.8.0, 2.7.3
• Hadoop Flags: Reviewed
• Assignee: Junping Du
• Reporter: Junping Du
• Created: 18/Apr/16 14:05
• Updated: 05/Jun/18 17:55
• Resolved: 21/Apr/16 02:04

Description

In our cluster based on a Cloud FileSystem, we notice JHS sometimes could skip directory with .jhist file in scanning.
The behavior is like:
First round scan, doesn't found .jhist file:

16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a directory with 6 files in it.
16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
...

Then, we see "Scan not needed of ..." for the same directory every 3 minutes until application failed as timeout.

From our analysis, we found the root cause is: most of Cloud File System (Azure FS, S3, etc.) is truncating file/directory modification time to seconds instead of milliseconds - which could due to limit of http protocol (from discussion at: https://forums.aws.amazon.com/thread.jspa?messageID=476615).

So if the time sequence is happen to be: latest non .jhist file modification on directory happens at T1, directory scanning happens at T2, .jhist file added to directory at T3. If we have T1< T2 < T3 and T1 is equal to T3 after truncating to seconds, this issue could appear.

Issue Links

• is related to: MAPREDUCE-7101 Add config parameter to allow JHS to alway scan user dir irrespective of modTime