[OOZIE-3566] Wrong numbers provided by Oozie monitoring webservices.requests sampler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.3.0
• Assignee: Maxim Kolesnikov
• Reporter: Maxim Kolesnikov
• Created: 05/Dec/19 14:15
• Updated: 09/Dec/19 16:10
• Resolved: 09/Dec/19 11:42

Description

webservices.requests sampler based on 

org.apache.oozie.servlet.JsonRestServlet#TOTAL_REQUESTS_SAMPLER_COUNTER

may produce wrong results if

org.apache.oozie.servlet.JsonRestServlet#service

rises an Exception during request URL validation.

 

TOTAL_REQUESTS_SAMPLER_COUNTER is incremented in try block and decremented in finally block right after that. But as validateRestUrl comes before the counter is incremented it may lead to situation when counter is decremented without being incremented before (in case of validation error).

 

Attached image is an example graph of webservices.requests being drawn in Grafana via Graphite. Pay attention to Y-axis. While most nodes publish positive numbers close to 0, some others seems to be shifted off by 1, with results close to -1, util two of these nodes jump to numbers close to -2 around 14:57. At this exact time both servers have URL validation errors in their logs, like this:

2019-11-29 14:57:29,394 WARN V2JobServlet:523 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[GET <SANITIZED>] error[E0301], E0301: Invalid resource [<SANITIZED>]

Issue Links

• links to: GitHub Pull Request #46