Chukwa
  1. Chukwa
  2. CHUKWA-506

Add retry logic to MetricDataLoader

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When processing data in Chukwa, many steps along the way, when the encounter an error condition, will attempt to reprocess the data a few times before giving up. This doesn't happen in post processing (at least when using MetricDataLoader); for example if there is a SQLException of any kind, it will move the data to the "inError" directory at which point it becomes AFAIK a manual process to move the data back to be reprocessed, so it would be nice to retry a few times in case the exception was transient (database was being bounced, or similar).

      I will attach a patch to be reviewed.

        Activity

        Hide
        William Bajzek added a comment -

        No problem. Thanks for committing it.

        • William Bajzek
          williambajzek@gmail.com
        Show
        William Bajzek added a comment - No problem. Thanks for committing it. William Bajzek williambajzek@gmail.com
        Hide
        Eric Yang added a comment -

        I just committed this, thanks William.

        Show
        Eric Yang added a comment - I just committed this, thanks William.
        Hide
        Eric Yang added a comment -

        Improve reliability for post Processor.

        Show
        Eric Yang added a comment - Improve reliability for post Processor.
        Hide
        Eric Yang added a comment -

        +1 Looks good.

        Show
        Eric Yang added a comment - +1 Looks good.
        Hide
        William Bajzek added a comment -

        A patch which implements this. I have tested it out by starting chukwa, shutting down MySQL, and then submitting a job. after the SQLException appeared in the postprocessor.log, I restarted MySQL and waited. The next time the postprocessor ran, the data was correctly inserted into the database.

        Unfortunately, I can't get the unit tests to pass even with an unchanged, freshly checked out copy of chukwa. They worked for me the other day, but not now, but I was told on chukwa-dev to submit the patch for review anyway.

        Show
        William Bajzek added a comment - A patch which implements this. I have tested it out by starting chukwa, shutting down MySQL, and then submitting a job. after the SQLException appeared in the postprocessor.log, I restarted MySQL and waited. The next time the postprocessor ran, the data was correctly inserted into the database. Unfortunately, I can't get the unit tests to pass even with an unchanged, freshly checked out copy of chukwa. They worked for me the other day, but not now, but I was told on chukwa-dev to submit the patch for review anyway.

          People

          • Assignee:
            Eric Yang
            Reporter:
            William Bajzek
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development