[SAMZA-1120] Config scope changes for multi-stage

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.13.0
• Fix Version/s: 0.15.0
• External issue ID: SAMZA-1041
• Assignee: Jake Maes
• Reporter: Jake Maes
• Created: 06/Mar/17 18:03
• Updated: 12/Jan/18 19:26

Description

With the multi-stage feature (SAMZA-1041), Samza will have the ability to run a collection of processors as a unit. To configure those processors with a single config, we need a way to independently configure each processor.

The current best idea is to introduce a new scope in the configs. Here are 2 options that differ only in naming.

1. Application->Job->Task scopes for configs. Application configs apply to the entire multistage application. A job config corresponds to a particular processor or job in the application and a task config applies to a particular task.
2. Job->Processor->Task scopes for configs. In this model, a Job is the deployable unit and corresponds to the full multistage application. A processor is an independent stage in the application.

The advantage of #1 is most developers seem to prefer the term "Application" over "Job"

The advantage of #2 is minimal renaming in the code and configs, which will likely make migration easier.

In both cases, we could define an inheritance structure s.t. any config not defined at the processor scope can be inherited from the application scope. This should reduce the verbosity of the configs.

btw. this is an issue that app.runner config with job.factory.class config. Do we allow configuring job.factory.class in different runner mode? If so, what are the options?

Issue Links

• is depended upon by: SAMZA-1041 Multi-stage feature for Samza