[CHUKWA-94] SALSA state-machine extraction from Hadoop logs

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Data Processors
• Assignee: Jiaqi Tan
• Reporter: Jiaqi Tan
• Created: 06/Apr/09 14:16
• Updated: 22/Sep/09 16:12

Description

This is a proposed feature addition to extract state-machine views from Hadoop's logs (TaskTracker, JobTracker, and DataNode currently supported, NameNode soon). These views are as described in http://www.usenix.org/event/wasl08/tech/full_papers/tan/tan_html/ and will enable analysis and diagnosis algorithms to be built on top of them.

Building a full SALSA view involves two steps:

1. Incrementally parsing log entries on a per-node basis to extract states (line-by-line reading, assuming the entire log file from a given node is available to the same process)
2. "Stitching" and correlating states across all logs (across nodes and across types) to build a full state machine.

My idea is to add SALSA as two jobs in the demux stage, with the first parsing job in demux, and either having:
(a) the parsing job write its output to the permanent store with the correlating job reading/writing from/to the permanent store, or
(b) the parsing job write its output back to the sinkfile and having the correlating job reading from the sink file and writing to the permanent store.

Issue Links

• incorporates: CHUKWA-360 Causal stitching of SALSA states
• incorporates: CHUKWA-344 State-Machine Generation for input to SALSA visualizations
• incorporates: CHUKWA-279 Swimlanes visualization for Hadoop job progress
• incorporates: CHUKWA-299 HDFS Spatial Heatmaps
• incorporates: CHUKWA-342 Static Swimlanes Visualization Widget
• incorporates: CHUKWA-343 Static HDFS Heatmap Visualization