Nutch
  1. Nutch
  2. NUTCH-1302

nutchgora job failures should be noticed by submitter

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I stumbled upon an issue where crawling seem to go right, only to notice much later on that jobs actually failed as a whole.

      This is caused because for most jobs that are submitted, Nutchgora does not check the 'succeeded' boolean that is returned. This should be done and acted upon appropriately. (Either throwing an exception or returning non-zero exit codes).

        Activity

        Hide
        Ferdy Galema added a comment -

        NutchJob is a nice wrapper of Hadoop's Job, so that I could override Job.waitForCompletion(verbose) in order to throw an exception whenever a job fails. (A job fails typically whenever one or more tasks have failed repeatedly.)

        Because some people may be reliant upon silent job failures (could not think of why, but just in case), I created an expert option that disables this behaviour, namely "fail.on.job.failure". When this property is set to false, jobs will fail silently like they did before.

        Show
        Ferdy Galema added a comment - NutchJob is a nice wrapper of Hadoop's Job, so that I could override Job.waitForCompletion(verbose) in order to throw an exception whenever a job fails. (A job fails typically whenever one or more tasks have failed repeatedly.) Because some people may be reliant upon silent job failures (could not think of why, but just in case), I created an expert option that disables this behaviour, namely "fail.on.job.failure". When this property is set to false, jobs will fail silently like they did before.
        Hide
        Hudson added a comment -

        Integrated in Nutch-nutchgora #185 (See https://builds.apache.org/job/Nutch-nutchgora/185/)
        NUTCH-1302 nutchgora job failures should be noticed by submitter (Revision 1297462)

        Result = SUCCESS
        ferdy :
        Files :

        • /nutch/branches/nutchgora/CHANGES.txt
        • /nutch/branches/nutchgora/src/java/org/apache/nutch/util/NutchJob.java
        Show
        Hudson added a comment - Integrated in Nutch-nutchgora #185 (See https://builds.apache.org/job/Nutch-nutchgora/185/ ) NUTCH-1302 nutchgora job failures should be noticed by submitter (Revision 1297462) Result = SUCCESS ferdy : Files : /nutch/branches/nutchgora/CHANGES.txt /nutch/branches/nutchgora/src/java/org/apache/nutch/util/NutchJob.java

          People

          • Assignee:
            Ferdy Galema
            Reporter:
            Ferdy Galema
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development