[OOZIE-3249] [tools] Instrumentation log parser

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.0.0
• Fix Version/s: 5.2.0
• Component/s: tools
• Assignee: Andras Piros
• Reporter: Andras Piros
• Created: 10/May/18 08:41
• Updated: 09/Dec/19 08:54
• Resolved: 05/Apr/19 07:05

Description

Oozie instrumentation logs contain a lot of information, but are difficult to parse, because per instrumentation log entry there is always one header line in plain text format (containing timestamp), and multiple other lines in JSON format (not containing timestamp). Those lines of course belong together.

2018-05-02 02:48:13,426  INFO oozieinstrumentation:520 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] 
{
...
  "counters" : {
...
    "callablequeue.executed" : {
      "count" : 5954144
    },
...
    "callablequeue.queued" : {
      "count" : 10596129
    },
...
  },
...
}

There should be a simple script in tools/bin that takes as parameters:

• input file name (-i), e.g. -i /path/to/oozie-instrumentation.log
• output file name (-o), e.g. -o /path/to/oozie-instrumentation.log.out
• parameters to extract (-p) in the format of path/to/json/value1,path/to/json/value2, in this case -p counters/callablequeue.executed/count,counters/callablequeue.queued/count

The output file should contain in CSV format:

• a header line containing column names for
• one line per parsed input header / JSON lines, containing:
  • first cell is the minutes part of the timestamp
  • consecutive cells are parsed JSON values given each parameter to extract