[GOBBLIN-742] Implement a FileSystem based JobConfigurationManager

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.15.0
• Component/s: gobblin-cluster
• Assignee: Hung Tran
• Reporter: Sudarshan Vasudevan
• Created: 18/Apr/19 21:05
• Updated: 07/Oct/20 19:13
• Resolved: 07/Oct/20 19:13

Description

This task aims to implement a FileSystem based JobConfigurationManager. The manager has an underlying SpecConsumer which reads specs from a config specified path on HDFS and persists them to a JobCatalog. On successful consumption of the spec by the manager , it is deleted from the original path.  So on the next fetch, only new specs will be picked up by the JobConfigurationManager.

Issue Links

• links to: GitHub Pull Request #2608