Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2951

Crawl datum with metadata WRITABLE_GENERATE_TIME_KEY awaits fetching forever

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.16, 1.17, 1.18
    • 1.19
    • generator
    • None

    Description

      When a crawl datum contains a WRITABLE_GENERATE_TIME_KEY metadata it will wait forever.

       

            LongWritable oldGenTime = (LongWritable) crawlDatum.getMetaData()
                .get(Nutch.WRITABLE_GENERATE_TIME_KEY);
            if (oldGenTime != null) { // awaiting fetch & update
              if (oldGenTime.get() + genDelay > curTime) // still wait for
                // update
                context.getCounter("Generator", "WAIT_FOR_UPDATE").increment(1);
              return;
            }

      The line 

      context.getCounter("Generator", "WAIT_FOR_UPDATE").increment(1); 

      has been introduced in version 1.16 without adding the brackets.

      Attachments

        Issue Links

          Activity

            People

              snagel Sebastian Nagel
              Lapax Lapadula Alessandro
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: