[NUTCH-3014] Standardize Job names

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.19
• Fix Version/s: 1.20
• Component/s: configuration, runtime
• Assignee: Lewis John McGibbney
• Reporter: Lewis John McGibbney
• Created: 21/Oct/23 06:01
• Updated: 03/Nov/23 04:39
• Resolved: 03/Nov/23 03:36

Description

There is a large degree of variability when we set the job name{}

 

Job job = NutchJob.getInstance(getConf());

job.setJobName("read " + segment);

 

Some examples mention the job name, others don't. Some use upper case, others don't, etc.

I think we can standardize the NutchJob job names. This would help when filtering jobs in YARN ResourceManager UI as well.

I propose we implement the following convention

• Nutch (mandatory) - static value which prepends the job name, assists with distinguishing the Job as a NutchJob and making it easily findable.
• ${ClassName} (mandatory) - literally the name of the Class the job is encoded in
• ${additional info} (optional) - value could further distinguish the type of job (LinkRank Counter, LinkRank Initializer, LinkRank Inverter, etc.)

Nutch ${ClassName}: ${additional info}

Examples:

• Nutch LinkRank: Inverter
• Nutch CrawlDb: + $crawldb
• Nutch LinkDbReader: + $linkdb

Thanks for any suggestions/comments.

Issue Links

• links to: GitHub Pull Request #789