[GEARPUMP-65] Propose to attach jar files in submitdag api

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: restapi
• Assignee: Unassigned
• Reporter: Huafeng Wang
• Created: 21/Apr/16 07:38
• Updated: 21/Apr/16 07:38

Description

This is issue imported from https://github.com/gearpump/gearpump/issues/1450

Currently to submit a dag, we need to submit corresponding jar files individually. If the application is terminated, the system does not have the obligation/knowledge to remove uploaded jars. We need to absorb uploadjar api and changesubmitdag, so that upload jar files and submit dag will be executed in one transaction.

The Design Thoughts
The current submitdag request has an application description (see SubmitApplicationRequest). Any processor in the description can be associated with an AppJar object, which consists of a form entity name and the file path to the uploaded file.

Now I propose to change submitdag as follows:
1. Move the application description into object app.
2. Any AppJar object will only have a form entity name, the file path will be empty.
3. The request can carry 0-n files. Their form entity name starts with "file_".

So backend will first save all jar files and then fill the file paths with the corresponding file paths. As a todo, app master should remember these temporary uploaded files and remove them, when the application is terminated.

Speaking to the dashboard UI part, when create a processor/edge, user need to have an extra step to choose a jar, before entering a task class or a partitioner class. If dashboard uses JsZip, we can reduce several steps.