Uploaded image for project: 'Chukwa (retired)'
  1. Chukwa (retired)
  2. CHUKWA-94

SALSA state-machine extraction from Hadoop logs

Add voteWatch issue
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Data Processors
    • None

    Description

      This is a proposed feature addition to extract state-machine views from Hadoop's logs (TaskTracker, JobTracker, and DataNode currently supported, NameNode soon). These views are as described in http://www.usenix.org/event/wasl08/tech/full_papers/tan/tan_html/ and will enable analysis and diagnosis algorithms to be built on top of them.

      Building a full SALSA view involves two steps:

      1. Incrementally parsing log entries on a per-node basis to extract states (line-by-line reading, assuming the entire log file from a given node is available to the same process)
      2. "Stitching" and correlating states across all logs (across nodes and across types) to build a full state machine.

      My idea is to add SALSA as two jobs in the demux stage, with the first parsing job in demux, and either having:
      (a) the parsing job write its output to the permanent store with the correlating job reading/writing from/to the permanent store, or
      (b) the parsing job write its output back to the sinkfile and having the correlating job reading from the sink file and writing to the permanent store.

      Attachments

        1. tan.pdf
          622 kB
          Jiaqi Tan

        Issue Links

          Activity

            People

              tanjiaqi Jiaqi Tan
              tanjiaqi Jiaqi Tan

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 672h
                  672h
                  Remaining:
                  Time Spent - 624h Remaining Estimate - 48h
                  48h
                  Logged:
                  Time Spent - 624h Remaining Estimate - 48h
                  624h

                  Slack

                    Issue deployment