[GOBBLIN-1087] Track and report histogram of observed lag from Gobblin Kafka pipeline

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.15.0
• Component/s: gobblin-kafka
• Assignee: Shirshanka Das
• Reporter: Sudarshan Vasudevan
• Created: 17/Mar/20 05:23
• Updated: 04/Apr/20 00:19
• Resolved: 04/Apr/20 00:19

Description

In this PR, we instrument the KafkaExtractor to track the observed latency of Kafka consumer records processed by the pipeline. Here, observed latency is measured as the time difference between processing time of the record and the original creation time. The latency distribution is tracked in an HdrHistogram, which is serialized into a string when emitted as part of a GobblinTrackingEvent.

Issue Links

• links to: GitHub Pull Request #2928