[GOBBLIN-1034] Ensure underlying writers are expired from the PartitionedDataWriter cache to avoid accumulation of writers for long running Gobblin jobs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.15.0
• Component/s: gobblin-core
• Assignee: Abhishek Tiwari
• Reporter: Sudarshan Vasudevan
• Created: 28/Jan/20 18:11
• Updated: 12/Feb/20 19:15
• Resolved: 12/Feb/20 19:15

Description

Currently, the underlying writers are never evicted from the PartitionedDataWriter cache. For long running Gobblin jobs (e.g. streaming), this will cause a memory leak particularly if the underlying writers maintain state. 

Issue Links

• links to: GitHub Pull Request #2876