[MAPREDUCE-778] [Rumen] Need a standalone JobHistory log anonymizer

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0, 2.0.0-alpha
• Fix Version/s: 0.23.1
• Component/s: tools/rumen
• Labels: anonymization rumen
• Hadoop Flags: Reviewed
• Release Note: Added an anonymizer tool to Rumen. Anonymizer takes a Rumen trace file and/or topology as input. It supports persistence and plugins to override the default behavior.
• Tags: rumen anonymization
• Assignee: Amar Kamat
• Reporter: Hong Tang
• Created: 21/Jul/09 07:42
• Updated: 10/Mar/15 04:31
• Resolved: 16/Dec/11 14:26

Description

Job history logs contain a rich set of information that can help understand and characterize cluster workload and individual job execution. Examples of work that parses or utilizes job history include HADOOP-3585, MAPREDUCE-534, HDFS-459, MAPREDUCE-728, and MAPREDUCE-776. Some of the parsing tools developed in previous work already contains a component to anonymize the logs. It would be nice to combine these effort and have a common standalone tool that can anonymizes job history logs and preserve much of the structure of the files so that existing tools on top of job history logs continue work with no modification.

Issue Links

• is related to: HADOOP-7470 move up to Jackson 1.8.8
• relates to: MAPREDUCE-3580 [Rumen] Rumen anonymizer should also parse and anonymize other job properties like 'mapreduce.output.fileoutputformat.compress.codec' etc
• relates to: MAPREDUCE-3581 [Rumen] Rumen anonymizer should handle composite string data