[SAMZA-2501] Optimizing startpoint manager to not make successive bootstrapMessage calls to coordinator-store

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5
• Assignee: Unassigned
• Reporter: Rayman
• Created: 01/Apr/20 22:34
• Updated: 27/May/20 03:38
• Resolved: 27/May/20 03:38

Description

Symptom: Currently the startpoint manager queries metastore for task-ssps and for each ssp queries coordinator-store, issuing a re-read on the coordinator store. This causes increased AM startup time and depending on the number of input SSPs to a job (thousands in case of regex), YARN may timeout the AM.
Cause: Above.
Fix: This change reduces to number of metastore reads to 2; one for startpoints keyed by ssp and the other for startpoints keyed by ssp+taskname.
API changes: None
Upgrade Instructions: None

Issue Links

• links to: GitHub Pull Request #1335