Chukwa
  1. Chukwa
  2. CHUKWA-344

State-Machine Generation for input to SALSA visualizations

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.2.0, 0.3.0
    • Fix Version/s: 0.3.0
    • Component/s: Data Processors
    • Labels:
      None
    • Release Note:
      MapReduce program to generate state-machine view from JobHistory and DataNode/TaskTracker ClientTrace logs

      Description

      MapReduce job that runs after Demux to generate State-Machine data structures used for visualization and other SALSA analysis.

      1. CHUKWA-344.patch
        68 kB
        Jiaqi Tan
      2. CHUKWA-344-1_with_unit_tests.patch
        165 kB
        Jiaqi Tan

        Issue Links

          Activity

          Hide
          Jiaqi Tan added a comment -

          All visualization widgets need input generated by this

          Show
          Jiaqi Tan added a comment - All visualization widgets need input generated by this
          Hide
          Jiaqi Tan added a comment -

          Upgraded to Blocker since you can't actually run the visualization without this. This is the last piece to the puzzle.

          Show
          Jiaqi Tan added a comment - Upgraded to Blocker since you can't actually run the visualization without this. This is the last piece to the puzzle.
          Hide
          Jiaqi Tan added a comment -

          State-machine generation + driver in bin/fsm.sh

          Show
          Jiaqi Tan added a comment - State-machine generation + driver in bin/fsm.sh
          Hide
          Ari Rabkin added a comment -

          I'm okay committing this. But reassure me that we don't need tests – do other components of SALSA come with test code that exercises the code in this patch?

          I'd rather that SALSA not rot, or at least not rot quickly, if you aren't around to maintain it.

          Show
          Ari Rabkin added a comment - I'm okay committing this. But reassure me that we don't need tests – do other components of SALSA come with test code that exercises the code in this patch? I'd rather that SALSA not rot, or at least not rot quickly, if you aren't around to maintain it.
          Hide
          Jiaqi Tan added a comment -

          Give me two days to think about testing

          Show
          Jiaqi Tan added a comment - Give me two days to think about testing
          Hide
          Jiaqi Tan added a comment -

          Will submit a new patch with test case, that should happen in about a week to two latest. Sorry for the delay.

          Show
          Jiaqi Tan added a comment - Will submit a new patch with test case, that should happen in about a week to two latest. Sorry for the delay.
          Hide
          Jiaqi Tan added a comment -

          Will add test cases later before I resubmit the complete patch (but if anyone is impatient enough, the patch works).

          Show
          Jiaqi Tan added a comment - Will add test cases later before I resubmit the complete patch (but if anyone is impatient enough, the patch works).
          Hide
          Ari Rabkin added a comment -

          In general, "resolution: later" means "not this version." Whereas we'd like to do SALSA before the next release. So marking issue as reopened, to make sure we don't lose track of it.

          Show
          Ari Rabkin added a comment - In general, "resolution: later" means "not this version." Whereas we'd like to do SALSA before the next release. So marking issue as reopened, to make sure we don't lose track of it.
          Hide
          Ari Rabkin added a comment -

          Having now looked more closely at this code –

          This looks like something very handy for ongoing Berkeley research in modeling Hadoop execution. What are your future plans for this code? If you don't intend to work on it further, I'm inclined to commit this, and then immediately get to work adding tests, comments, and doing a bit of refactoring to make it useful for my purposes.

          But it's your code, so I don't want to hack at it if you have concrete plans. So let me know.

          Show
          Ari Rabkin added a comment - Having now looked more closely at this code – This looks like something very handy for ongoing Berkeley research in modeling Hadoop execution. What are your future plans for this code? If you don't intend to work on it further, I'm inclined to commit this, and then immediately get to work adding tests, comments, and doing a bit of refactoring to make it useful for my purposes. But it's your code, so I don't want to hack at it if you have concrete plans. So let me know.
          Hide
          Jiaqi Tan added a comment -

          Yes there are future plans--immediate plan is to write a full unit test that begins from job history logs. I should have that code in by middle next week, so this patch will be complete with tests. Later plans are to build more complex models, based on the generated data.

          What purposes did you have in mind?

          Show
          Jiaqi Tan added a comment - Yes there are future plans--immediate plan is to write a full unit test that begins from job history logs. I should have that code in by middle next week, so this patch will be complete with tests. Later plans are to build more complex models, based on the generated data. What purposes did you have in mind?
          Hide
          Ari Rabkin added a comment -

          I wanted to stitch together the state machine transitions into a full causal graph of the job execution, and then build some kind of mechanized analysis for finding bottlenecks and tracking them over time – figuring out if you have a slow task or a slow node or what.

          Show
          Ari Rabkin added a comment - I wanted to stitch together the state machine transitions into a full causal graph of the job execution, and then build some kind of mechanized analysis for finding bottlenecks and tracking them over time – figuring out if you have a slow task or a slow node or what.
          Hide
          Ari Rabkin added a comment -

          A thing that would be useful to me, and probably good in general, would be a paragraph or two of documentation describing the file formats that your code expects and outputs.

          Show
          Ari Rabkin added a comment - A thing that would be useful to me, and probably good in general, would be a paragraph or two of documentation describing the file formats that your code expects and outputs.
          Hide
          Jiaqi Tan added a comment -

          The stitching has already been done, I have a pig script that does that and it's described in our HotCloud paper as well.

          Sure I can work on the documentation for this patch as well, together with the unit tests.

          Show
          Jiaqi Tan added a comment - The stitching has already been done, I have a pig script that does that and it's described in our HotCloud paper as well. Sure I can work on the documentation for this patch as well, together with the unit tests.
          Hide
          Ari Rabkin added a comment -

          Awesome. If you're able to release it, that script would be a good fit for chukwa/contrib.

          Show
          Ari Rabkin added a comment - Awesome. If you're able to release it, that script would be a good fit for chukwa/contrib.
          Hide
          Jiaqi Tan added a comment -

          FSMBuilder with unit tests

          Show
          Jiaqi Tan added a comment - FSMBuilder with unit tests
          Hide
          Jiaqi Tan added a comment -

          Added unit tests to test for JobHistory and ClientTrace message-generated state-machine entries

          Show
          Jiaqi Tan added a comment - Added unit tests to test for JobHistory and ClientTrace message-generated state-machine entries
          Hide
          Ari Rabkin added a comment -

          I vote to commit this, and will do so later today barring objection.

          Show
          Ari Rabkin added a comment - I vote to commit this, and will do so later today barring objection.
          Hide
          Ari Rabkin added a comment -

          I just committed this. Thanks, Jiaqi!

          Show
          Ari Rabkin added a comment - I just committed this. Thanks, Jiaqi!
          Hide
          Hudson added a comment -

          Integrated in Chukwa-trunk #142 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/142/)
          . State-Machine Generation for input to SALSA visualizations. Contributed by Jiaqi Tan

          Show
          Hudson added a comment - Integrated in Chukwa-trunk #142 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/142/ ) . State-Machine Generation for input to SALSA visualizations. Contributed by Jiaqi Tan

            People

            • Assignee:
              Jiaqi Tan
              Reporter:
              Jiaqi Tan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development