Details
-
Improvement
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
In some heavy computation clusters, user may continually submit lots of jobs, in our scenario, there are 240k jobs per day. On average, 5 nodes will participate in running a job. All these job's log file will be aggregated on the hdfs. That is a big load for namenode. The total number of generated log files in the default cleaning period (1 week) can be calculated as follows:
AM logs per week: 7 days * 240,000 jobs/day * 2 files/job = 3360,000 files
App logs per week: 7 days * 240,000 jobs/day * 5 nodes/job * 1 file/node = 8400,000 files
There will be more than 10 million log files generated in one week. Even worse, some environments have to keep the logs for potential issues tracking for longer time. In general, these small log files will occupy about 12G heap size of Namenode, and impact the response speed of Namenode.
For optimizing the log management of history server, the main goals are:
1) Reduce the total count of files in HDFS.
2) Compatible with the former history server operation.
As per the goals above, we can mine the detail demands as follows:
1) Merge log files into bigger ones in HDFS periodically.
2) Optimized design should inherits from the original architecture to make the merged logs transparent to be browsed.
3) Merged logs should be aged periodically just like the common logs.
The whole life cycle of the AM logs:
1.Created by Application Master in intermediate-done-dir.
2.Moved to done-dir after the job is done.
3.Archived to archived-dir periodically.
4.Cleaned when all the logs in harball are expired.
The whole life cycle of the App logs:
1.Created by Applications in local-dirs.
2.Aggregated to remote-app-log-dir after the job is done.
3.Archived to archived-dir periodically.
4.Cleaned when all the logs in harball are expired.
