Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-155

Job History status arrive out of order causing the status to update incorrectly.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.2
    • Labels:
      None
    • Environment:

      Redhat 5.1, Java 6

      Description

      Job history contains lines like:

      Job JOBID="job_200903310541_1747" JOB_STATUS="RUNNING" .
      ...
      Job JOBID="job_200903310541_1747" FINISH_TIME="1238542231308" JOB_STATUS="SUCCESS" FINISHED_MAPS="1338" FINISHED_REDUCES="760" FAILED_MAPS="78" FAILED_REDUCES="43" COUNTERS="..." .

      When pushing the data through collectors and demux, the data can arrive out of order. The database is updated with status "RUNNING" instead of "SUCCESS".

      Chukwa Sequence ID can be used to sort out of order data before the data is pumped to database.

        Activity

        Hide
        zhangyongjiang Cheng added a comment -

        We could ask Hadoop team to add time stamp. But as a generic data collecting system, we should have the ability to deal with any kind of source data.

        Show
        zhangyongjiang Cheng added a comment - We could ask Hadoop team to add time stamp. But as a generic data collecting system, we should have the ability to deal with any kind of source data.
        Hide
        jboulon Jerome Boulon added a comment -

        +1 on asking Hadoop team to add time stamp since we want to do some time based analytic.

        Demux is able to deal with any kind of data but if there's some rules.
        It's the parser responsibility to provide

        • provide a time stamp, if any, use the default one provided by the Collector at the Chunk level
        • a key that will group information together according to the data usage

        Regarding the case where the data does not contain any time stamp the system will do a best effort to partition the data based on collector time stamp but the parser could/should guarantee the order by specifying a key that contains the SeqId + offset within the same chunk.

        Show
        jboulon Jerome Boulon added a comment - +1 on asking Hadoop team to add time stamp since we want to do some time based analytic. Demux is able to deal with any kind of data but if there's some rules. It's the parser responsibility to provide provide a time stamp, if any, use the default one provided by the Collector at the Chunk level a key that will group information together according to the data usage Regarding the case where the data does not contain any time stamp the system will do a best effort to partition the data based on collector time stamp but the parser could/should guarantee the order by specifying a key that contains the SeqId + offset within the same chunk.
        Hide
        zhangyongjiang Cheng added a comment -

        Added timestamp for job log.

        If JOB_STATUS == "SUCCESS", ChukwaRecord will have JOB_STATUS_SUCCESS="timestamp"
        If JOB_STATUS == "PREP", ChukwaRecord will have JOB_STATUS_PREP="timestamp"
        ......
        If JOB_STATUS == null, ChukwaRecord will have JOB_STATUS_null="timestamp"

        Show
        zhangyongjiang Cheng added a comment - Added timestamp for job log. If JOB_STATUS == "SUCCESS", ChukwaRecord will have JOB_STATUS_SUCCESS="timestamp" If JOB_STATUS == "PREP", ChukwaRecord will have JOB_STATUS_PREP="timestamp" ...... If JOB_STATUS == null, ChukwaRecord will have JOB_STATUS_null="timestamp"
        Hide
        jboulon Jerome Boulon added a comment -

        my 2 cents ...

        Or we could have a more generic way:

        The problem is that the mysql Job table does not contains columns for more than one state.
        What we care about in this table is the final Job state.
        So why not create one additional key JOB_FINAL_STATE that will contain the final Job state, aka, success, killed or failed.
        the "JOB_FINAL_STATE" 's key could be created at the same time as the finish-time parsing from JobHistory.
        For example:
        Job JOBID="job_200903310541_1200" FINISH_TIME="1238528943585" JOB_STATUS="SUCCESS" will give JOB_FINAL_STATE="SUCCESS"

        Then all others JOB_STATUS should remain unchanged, aka JOB_STATUS="WHAT_EVER_THE_VALUE_IS", timestamp will be what ever is available at that time.
        This will give us the transition-states table.

        Show
        jboulon Jerome Boulon added a comment - my 2 cents ... Or we could have a more generic way: The problem is that the mysql Job table does not contains columns for more than one state. What we care about in this table is the final Job state. So why not create one additional key JOB_FINAL_STATE that will contain the final Job state, aka, success, killed or failed. the "JOB_FINAL_STATE" 's key could be created at the same time as the finish-time parsing from JobHistory. For example: Job JOBID="job_200903310541_1200" FINISH_TIME="1238528943585" JOB_STATUS="SUCCESS" will give JOB_FINAL_STATE="SUCCESS" Then all others JOB_STATUS should remain unchanged, aka JOB_STATUS="WHAT_EVER_THE_VALUE_IS", timestamp will be what ever is available at that time. This will give us the transition-states table.
        Hide
        eyang Eric Yang added a comment -

        +1 with Jerome's proposal.

        Show
        eyang Eric Yang added a comment - +1 with Jerome's proposal.
        Hide
        zhangyongjiang Cheng added a comment -

        new patch submitted. if finish_time found, add a new key JOB_FINAL_STATUS.

        Show
        zhangyongjiang Cheng added a comment - new patch submitted. if finish_time found, add a new key JOB_FINAL_STATUS.
        Hide
        eyang Eric Yang added a comment -

        +1 looks good.

        Show
        eyang Eric Yang added a comment - +1 looks good.
        Hide
        eyang Eric Yang added a comment -

        I just committed this, thanks Cheng.

        Show
        eyang Eric Yang added a comment - I just committed this, thanks Cheng.
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #8 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/8/ )
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #45 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/45/ )

          People

          • Assignee:
            zhangyongjiang Cheng
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development