[GOBBLIN-202] Add better metrics to gobblin to support AWS autoscaling

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Sprint: Apache Gobblin 170807
• Assignee: Joel Baranick
• Reporter: Joel Baranick
• Created: 10/Aug/17 16:11
• Updated: 03/Jan/18 06:50
• Resolved: 15/Aug/17 19:31

Description

Autoscaling Gobblin via CPU usage is suboptimal as it is not related to the amount of work to do. In a perfect system gobblin worker nodes would be running at nearly 100% cpu. Therefore, autoscaling by this metric would not work. To support autoscaling new metrics should be added that indicate the amount of work that gobblin needs to do.