Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.7.0
-
None
-
None
Description
cc4e3957ba29bb42112dc21bfa8242e3f7afeac6 introduced the kudu-log-parser.pl script, which takes a collection of possibly-gzipped Kudu logs, categorizes and extracts information from some events in the logs using regexes, and then sorted-merges all the logs together. It can be pretty useful for looking at problems in a Kudu cluster ex post facto, especially when the exact timeframe or cause is not known.
There's a number of things that can be done to make the script better, including:
1. Eliminating or disambiguating some false matches, e.g. "Time spent" is a prefix matched on that applies both to slow execution logging and to LBM startup messages.
2. Parallelizing the processing. In my experience, the script can take 30 minutes to munch a 12-node cluster's logs if the logs are 100-200MB in size.
3. Mike wrote the script to look at a cluster with consensus issues, so most of the categorization if focused on those types of logs. We cold generalize it to more types, and also allow filtering based on types.
4. The script is written in Perl. While that language is dear to Mike, most Kudu developers would be more comfortable using and tweaking the script if it were written in a more widely-known language like Python. Of course, Cython doesn't support parallelism, so maybe something like Scala? That has more unusual prerequisites, but it's Java-like and can be run as a script.