Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-430

Narrow down input for FSM mapreduce job

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.4.0
    • Component/s: MR Data Processors
    • Labels:
      None

      Description

      FSMDataloader supplies all demux output data to FSM state machine. This is not efficient because most of the data type do not contribute to state generation. According to Jiaqi, the state machine requires the following types:

      JobHistoryTaskDataMapper:

      /chukwa/repos/chukwa/JobData
      /chukwa/repos/chukwa/TaskData

      TaskTrackerClientTraceMapper:

      /chukwa/repos/chukwa/ClientTraceDetailed

      DataNodeClientTraceMapper:

      /chukwa/repos/chukwa/ClientTraceDetailed

      This jira is to optimize the data loader supplied input, and narrow down the required input type.

        Activity

        Hide
        eyang Eric Yang added a comment -

        Narrow down the input type from 70 to 4.

        Show
        eyang Eric Yang added a comment - Narrow down the input type from 70 to 4.
        Hide
        asrabkin Ari Rabkin added a comment -

        +1 to patch

        Show
        asrabkin Ari Rabkin added a comment - +1 to patch
        Hide
        tanjiaqi Jiaqi Tan added a comment -

        +1 to patch, looks good to me

        Show
        tanjiaqi Jiaqi Tan added a comment - +1 to patch, looks good to me
        Hide
        eyang Eric Yang added a comment -

        I just committed this, thanks Ari and Jiaqi.

        Show
        eyang Eric Yang added a comment - I just committed this, thanks Ari and Jiaqi.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #229 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/229/)
        . Narrow down the list of demux output for FSM to improve processing time. (Eric Yang)

        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #229 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/229/ ) . Narrow down the list of demux output for FSM to improve processing time. (Eric Yang)

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development