[NUTCH-2951] Crawl datum with metadata WRITABLE_GENERATE_TIME_KEY awaits fetching forever

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.16, 1.17, 1.18
• Fix Version/s: 1.19
• Component/s: generator
• Assignee: Sebastian Nagel
• Reporter: Lapadula Alessandro
• Created: 31/May/22 13:49
• Updated: 13/Mar/24 14:51
• Resolved: 21/Jun/22 10:37

Description

When a crawl datum contains a WRITABLE_GENERATE_TIME_KEY metadata it will wait forever.

 

      LongWritable oldGenTime = (LongWritable) crawlDatum.getMetaData()
          .get(Nutch.WRITABLE_GENERATE_TIME_KEY);
      if (oldGenTime != null) { // awaiting fetch & update
        if (oldGenTime.get() + genDelay > curTime) // still wait for
          // update
          context.getCounter("Generator", "WAIT_FOR_UPDATE").increment(1);
        return;
      }

The line 

context.getCounter("Generator", "WAIT_FOR_UPDATE").increment(1); 

has been introduced in version 1.16 without adding the brackets.

Issue Links

• is caused by: NUTCH-2737 Generator: count and log reason of rejections during selection
• links to: GitHub Pull Request #732