Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
Looking to reconstruct a timeline from write of recovered.edits file back to start of WAL file split, with a bunch of unrelated activity in the meantime, there isn't a consistent token that links split file write messages (which include store path including region hash) to beginning of WAL splitting activity. Sessonizing by host doesn't work because work can bounce around through retries. Thread context names in the logs vary and can be like [nds1-225-fra:60020-7] or [fb472085572ba72e96f1] (trailing digits of region hash) or [splits-1589016325868] .
We could have WALSplitter get the current time when starting the split of a WAL file and have it log this timestamp in every line as a splitting session identifier.
Related, we should track the time of split task execution end to end and export a metric that captures it.
It might also be worthwhile to wire up more of WAL splitting to TaskMonitor status logging. If we do this we can also enable status journal logging, so when a WAL split has completed, a line will appear in the log that has the list of all status messages recorded during splitting and the time delta in milliseconds between them.