[NUTCH-2481] HostDatum deltas(previous step statistics) and Metadata expressions

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.15
• Component/s: hostdb
• Assignee: Unassigned
• Reporter: Semyon Semyonov
• Created: 15/Dec/17 12:58
• Updated: 15/Nov/19 10:47

Description

To allow the usage of previous step statistics(deltas of fetched,unfetced etc) in hostdb. The motivation is usage of this statistics in generate with maxCount expressions.

 

The solution allows to fill in metadata of hostdatum based on custom JEXL expression using two hostdatum: before update(previousHostDatum) and after update(currentHostDatum)..

For example to fill in difference in quantity of fetched at round t and t-1 we can use the following expression

<property>
<name>hostdb.deltaExpression</name>
<value>{return new ("javafx.util.Pair","FetchedDelta", currentHostDatum.fetched - previousHostDatum.fetched);}</value>
</property>

A pull request will be provided shortly.

Issue Links

• Parent Feature: NUTCH-2368 Variable generate.max.count and fetcher.server.delay
• links to: GitHub Pull Request #278
• links to: Pull request