Description
Samza's existing config setup is problematic for a number of reasons:
- It's completely immutable once a job starts. This prevents any dynamic reconfiguration and auto-scaling. It is debatable whether we want these feature or not, but our existing implementation actively prevents it. See SAMZA-334 for discussion.
- We pass existing configuration through environment variables. YARN exports environment variables in a shell script, which limits the size to the varargs length on the machine. This is usually ~128KB. See SAMZA-333 and
SAMZA-337for details. - User-defined configuration (the Config object) and programmatic configuration (checkpoints and TaskName:State mappings (see
SAMZA-123)) are handled differently. It's debatable whether this makes sense.
In SAMZA-123, jghoman and I propose implementing a ConfigLog. This log would replace both the checkpoint topic and the existing config environment variables in SamzaContainer and Samza's YARN AM.
I'd like to keep this ticket's scope limited to just the implementation of the ConfigLog, and not re-designing how Samza's config is used in the code (SAMZA-40). We should, however, discuss how this feature would affect dynamic reconfiguration/auto-scaling.
Attachments
Attachments
Issue Links
- incorporates
-
SAMZA-798 Performance and stability issue after combining checkpoint and coordinator stream
- Resolved
- is related to
-
SAMZA-40 Refactor Samza configuration
- Open
-
SAMZA-42 Add a job setup phase to Samza
- Open
-
SAMZA-333 Large samza configurations results in yarn job failure
- Open
-
SAMZA-374 Need to be able to change SSP Grouper
- Resolved
-
SAMZA-406 Hot standby containers
- Open
- supercedes
-
SAMZA-237 Consider implementing job control topic to support dynamic inputs, capacity changes, etc.
- Resolved
1.
|
Pass config via HTTP | Resolved | Chris Riccomini | |
2.
|
Provide a Samza job data model for job coordinator | Resolved | Chris Riccomini | |
3.
|
Pass config from JobRunner to JobCoordinator via ConfigStream | Resolved | Chris Riccomini | |
4.
|
Use coordinator stream and eliminate CheckpointManager | Resolved | Naveen Somasundaram | |
5.
|
Document coordinator stream | Resolved | Navina Ramesh | |
6.
|
Migrate checkpoint from checkpoint topic to Coordinator stream | Resolved | Yi Pan | |
7.
|
Disable checkpoint in coordinator stream | Resolved | Yi Pan | |
8.
|
Integrate CoordinatorStream to use SystemConsumers and SystemProducers | Open | Unassigned | |
9.
|
Optimize CoordinatorStream's bootstrap mechanism | Open | Unassigned | |
10.
|
Refactor Coordinator stream messages | Resolved | József Márton Jung | |
11.
|
Explicit restart containers to pick up dynamic JobModel changes | Open | Alex Buck |